butspeechfit/vbx

Variational Bayes HMM over x-vectors diarization

VBHMM x-vectors Diarization (aka VBx)

Diarization recipe for CALLHOME, AMI and DIHARD II by Brno University of Technology.
The recipe consists of

• computing x-vectors
• doing agglomerative hierarchical clustering on x-vectors as a first step to produce an initialization
• apply variational Bayes HMM over x-vectors to produce the diarization output
• score the diarization output

More details about the full recipe in
F. Landini, J. Profant, M. Diez, L. Burget: Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks

If you are interested in the original version of VBx (prepared for the Second DIHARD Challenge), please refer to the corresponding branch.
If you are interested in the VBx recipe prepared for the track 4 of VoxSRC-20 Challenge (on VoxConverse), please refer to the corresponding branch.

Usage

To run the recipe, execute the run scripts for the different datasets with the corresponding parameters. Please refer to the scripts for more details. The CALLHOME and DIHARD II recipes require the corresponding datasets and the paths need to be provided. For AMI, the recordings need to be downloaded (for free) but the VAD segments and reference rttms are obtained from our proposed setup.

This repository has x-vector extractors already trained to function as a standalone recipe. However, the recipes for training the extractors can be found here.

The recipe as presented runs AHC as initialization for VBx. However, running AHC on long files (more than 30 minutes) can become very slow. We have added another type of initialization: random_ which means assigning random speaker labels as initialization and run VBx. This method runs substantially faster on long recordings but can have a slightly worse performance than initializing with AHC.

Getting started

We recommend to create anaconda environment

conda create -n VBx python=3.9
conda activate VBx

Clone the repository

git clone https://github.com/BUTSpeechFIT/VBx.git

Install the package

pip install -e .

Initialize submodule dscore:

git submodule init
git submodule update

Run the example

./run_example.sh

The output (last few lines) should look like this

File               DER    JER    B3-Precision    B3-Recall    B3-F1    GKT(ref, sys)    GKT(sys, ref)    H(ref|sys)    H(sys|ref)    MI    NMI
---------------  -----  -----  --------------  -----------  -------  ---------------  ---------------  ------------  ------------  ----  -----
ES2005a           7.06  29.99            0.65         0.78     0.71             0.71             0.56          1.14          0.59  1.72   0.67
*** OVERALL ***   7.06  29.99            0.65         0.78     0.71             0.71             0.56          1.14          0.59  1.72   0.67

Citations

In case of using the software please cite:
F. Landini, J. Profant, M. Diez, L. Burget: Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks (arXiv version)

@article{landini2022bayesian,
  title={Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks},
  author={Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
  journal={Computer Speech \& Language},
  volume={71},
  pages={101254},
  year={2022},
  publisher={Elsevier}
}

Results

We present here the results of our systems for the different datasets and different evaluation protocols. A more thorough discussion on the protocols and results can be found in the paper. All results are obtained using oracle VAD.

CALLHOME                         DIHARD II
-----------------                -------------------------
Protocol    DER                  Protocol        DER
Forgiving   4.42	                      dev    eval
Fair        14.21                Full        18.19   18.55
Full        21.77                Fair        12.23   12.29

AMI beamformed                   AMI Mix-Headset
-------------------------        -------------------------
Protocol         DER             Protocol         DER 
             dev     eval                     dev     eval
Forgiving    2.80    3.90        Forgiving    1.56    2.10
Fair        10.81   14.23        Fair         9.68   12.53
Full        17.66   20.84        Full        16.33   18.99

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contact

If you have any comment or question, please contact [email protected] or [email protected]

vbx's Issues

How to train the xvector extractor with own data

Hello，
I tried to use VBx-training-recipe to train resnet101 xvector extractor with cnceleb1 and cnceleb2. After 210 epochs training, I replaced the raw_75.pth, but got bad result. I found that the fbank of VBx-training-recipe was extracted by kaldi, that of VBx was extracted by features.py.
What I'm not sure whether it is right to use VBx-training-recipe directly to trian xvector extractor. How do I train the xvector extractor with own data？

feature diff vs kaldi

now i want to check the train scripts and test scripts.when i check the fbank64.i find some diff in featurs.py and kaldi fbank conf.
i do not know something wrong with the diff.
thank you

use pytorch+GPU but got poor performance

Hi,
I tested many datasets such as CALLHOME, AMI with onnx backend and CPU, and everything went well. When I turned to pytorch backend with GPU, however, the diarization result seemed to have something wrong.
I made some change on run_example.sh to run with pytorch and GPU:

      # run feature and x-vectors extraction
      python VBx/predict.py --in-file-list exp/list.txt \
          --in-lab-dir example/vad \
          --in-wav-dir example/audios/16k \
          --out-ark-fn exp/${filename}.ark \
          --out-seg-fn exp/${filename}.seg \
          --weights VBx/models/ResNet101_16kHz/nnet/raw_81.pth \
          --backend pytorch \
          --model ResNet101 \
          --gpus 7

and the output showed as follows:

nohup: ignoring input
INFO:__main__:Using GPU: 7
INFO:__main__:Start: Processing file ES2005a: 
INFO:__main__:End:   Processing file ES2005a: Elapsed: 38.42987585067749 seconds
ES2005a
Loading speaker turns from reference RTTMs...
Loading speaker turns from system RTTMs...
WARNING: No universal evaluation map specified. Approximating from reference and speaker turn extents...
Trimming reference speaker turns to UEM scoring regions...
Trimming system speaker turns to UEM scoring regions...
Checking for overlapping reference speaker turns...
Checking for overlapping system speaker turns...
Scoring...
File               DER    JER    B3-Precision    B3-Recall    B3-F1    GKT(ref, sys)    GKT(sys, ref)    H(ref|sys)    H(sys|ref)    MI    NMI
---------------  -----  -----  --------------  -----------  -------  ---------------  ---------------  ------------  ------------  ----  -----
ES2005a          52.46  86.04            0.31         1.00     0.47             0.99             0.15          2.35          0.01  0.51   0.42
*** OVERALL ***  52.46  86.04            0.31         1.00     0.47             0.99             0.15          2.35          0.01  0.51   0.42

diarization result is as below, and it seemed there were totally no speaker changes, which confused me so much.

SPEAKER ES2005a 1 0.000000 25.150000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 25.520000 8.970000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 35.900000 1.550000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 38.330000 2.070000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 40.420000 66.280000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 106.710000 26.950000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 133.880000 2.380000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 137.190000 5.570000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 143.040000 36.370000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 180.430000 17.960000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 199.710000 0.330000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 201.710000 12.500000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 214.860000 0.300000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 215.210000 8.800000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 225.080000 1.920000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 228.960000 12.020000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 241.370000 0.960000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 243.630000 4.890000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 250.200000 1.590000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 253.100000 1.550000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 255.080000 0.980000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 266.840000 2.430000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 276.780000 6.410000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 283.870000 17.930000 <NA> <NA> 1 <NA> <NA>
SPEAKER ES2005a 1 302.140000 4.450000 <NA> <NA> 1 <NA> <NA>

I also tried to tune some hyperparameters but it didn't work. Is there something wrong with my changes on the shell script, the weight checkpoint, or the code?

Ark file

Do you have the example ark file avaliable so i can cross reference it against what i am producing? Also did you convert these ark files into csv files at all? If so should there be 256 rows worth of tuples per vector?

Great project and ive had some good fun playing around with it.

Kind regards

using ground-truth VAD

Hi,

Nice job ! Thanks for sharing.

I am trying to use your code to run VoxConverse. I tried to use ground-truth VAD results for clustering, but I found that VAD results in VBx/VAD/final_system is not consistent with the officially released ones. For example, an official VAD result (here) has more segments than yours (here)

How can I solve this ? Thanks.

Speaker loop elimination

Hi guys I have a small update for your main VB_diarization function. This update almost fully removes your inner speaker loop and also speeds up things a little bit. I tried to make a pull request, but haven't found an opportunity to push new branch, anyway I suggest replacing the following lines:

for ii in range(maxIters):
    L = 0 # objective function (37) (i.e. VB lower-bound on the evidence)
    Ns = gamma.sum(0)                                     # bracket in eq. (34) for all 's'
    VtiEFs = gamma.T.dot(VtiEF)                           # eq. (35) except for \Lambda_s^{-1} for all 's'
    for sid in range(maxSpeakers):
        invL = np.linalg.inv(np.eye(R) + Ns[sid]*VtiEV*Fa/Fb) # eq. (34) inverse
        a = invL.dot(VtiEFs[sid])*Fa/Fb                                        # eq. (35)
        # eq. (29) except for the prior term \ln \pi_s. Our prior is given by HMM
        # trasition probability matrix. Instead of eq. (30), we need to use
        # forward-backwar algorithm to calculate per-frame speaker posteriors,
        # where 'lls' plays role of HMM output log-probabilities
        lls[:,sid] = Fa * (G + VtiEF.dot(a) - 0.5 * ((invL+np.outer(a,a)) * VtiEV).sum())
        L += Fb* 0.5 * (logdet(invL) - np.sum(np.diag(invL) + a**2, 0) + R)

with the following code:

for ii in range(maxIters):
    L = 0 # objective function (37) (i.e. VB lower-bound on the evidence)
    Ns = np.sum(gamma, axis=0)[:, np.newaxis, np.newaxis]               # bracket in eq. (34) for all 's'
    VtiEFs = gamma.T.dot(VtiEF)[:, :, np.newaxis]                       # eq. (35) except for \Lambda_s^{-1} for all 's'
    invLs = np.linalg.inv(np.eye(R)[np.newaxis, :, :] + Ns * VtiEV[np.newaxis, :, :] * Fa / Fb)  # eq. (34) inverse
    a = np.matmul(invLs, VtiEFs).squeeze(axis=-1) * Fa / Fb  # eq. (35)
    # eq. (29) except for the prior term \ln \pi_s. Our prior is given by HMM
    # trasition probability matrix. Instead of eq. (30), we need to use
    # forward-backwar algorithm to calculate per-frame speaker posteriors,
    # where 'lls' plays role of HMM output log-probabilities
    lls = Fa * (
            G[:, np.newaxis] + VtiEF.dot(a.T) - 0.5 *
            ((invLs + np.matmul(a[:, :, np.newaxis], a[:, np.newaxis, :])) * VtiEV[np.newaxis, :, :]).sum(axis=(1, 2)))
    for sid in range(maxSpeakers):
        L += Fb* 0.5 * (logdet(invLs[sid]) - np.sum(np.diag(invLs[sid]) + a[sid]**2, 0) + R)

as you can see I haven't implemented vectorized form for the last line in the speaker loop because this replacement requires additional modification of the logdet function. To make things work for multiple matrices spl.cholesky should be replaced with np.linalg.cholesky and for some reason this slows down the processing speed (probably because numpy version involves some additional checks before doing Cholesky decomposition). Finally lls = np.zeros_like(gamma) can be removed. What do you think about it?

Number of CPU cores as argument

Hello,

I noticed that "run_example.sh" uses all the cores of the machine. I looked through the code, but it doesn't seem very obvious to me how I could give as a parameter the maximum number of cores that will be used for "VBx/predict.py" and "VBx/vbhmm.py".
I'm interested in this to evaluate the processing time related to the available hardware resources.
So, is there a way to set the number of CPUs that will be used?

Thanks!

PermissionError with scorelib

Unable to complete scoring step:

PermissionError: [Errno 13] Permission denied: '/dscore/scorelib/md-eval-22.pl'

FIXED: I had to change to executable mode for the Perl file.

Overlapping speech

Hello,

What about overlapping segments with two (or more) people speaking at the same time ?
Is it handled by this solution (it seems it is not but I may have misunderstood something) ?
If not, do you have ideas on how to tackle this issue ?

Regards,

Yannis

How to run VBx on other dataset?

Hi there,
I am very interested in VBx, and I want to run VBx on other datasets to get the results of the speaker diarization.
I want to know what to prepare, or what code to run to achieve the above task?
Thank you very much for your reply.

Documentation for using the pre-trained model?

what does out_labels represents?
How can pretrained model used to get diarization outputs? Is there any doucmentation,
Model is executing well on the example given in repo.
How to generated rttm files?

The parameters about overlap in ur paper

Hello:
I have read the paper " ANALYSIS OF THE BUT DIARIZATION SYSTEM FOR VOXCONVERSE CHALLENGE", and gain lots of experience. I see the setences that "we tuned the detection threshold on Vox Converse development in order to maximize the precision." I try to use pyannote to detect overlap with default parameters but I dont get a better result than u. Could u share the best parameters for me, it will be helpful for me!! Thanks a lot!

Inconsistency between the run_example.sh and vbhmm.py

Hi, Fede,
Hope you are doing well, I tried to run your scripts for voxconverse but didn't make it, the main problem is

          python VBx/vbhmm.py --init AHC+VB \
          --out-rttm-dir exp \
          --xvec-ark-file exp/${filename}.ark \   
          --segments-file exp/${filename}.seg \  
          --xvec-transform VBx/models/ResNet101_16kHz/transform.h5 \   # this arg is depreceted in vbhmm.py
          --plda-file VBx/models/ResNet101_16kHz/plda \
          --threshold -0.015 \
          --lda-dim 128 \
          --Fa 0.3 \
          --Fb 17 \
          --loopP 0.99, 

          --xvec-mean # this is required in vbhmm.py but not provided in the shell script, 
                     # also xvec-mean didn't apear in the directory of VBx/models/ResNet101_16kHz

It would be great if you can fix this!
Best,
Shuai

Same ClusterID/SpeakerID for the incoming files

Hello Federico,

Can you please suggest if we can in any way keep the clusterID/speakerID same for the incoming short clips from the same audio. Actually i am trying to implement it for the streaming mode, wherein we get the short chunks of audio/of people speaking on the microphone. However for every new chunk the speaker id is new even though the speakers are actually the same. e.g. If only 1 speaker is actually speaking then every 10 seconds new chunk will be received and every time the cluster id or the speaker id generated will be different giving the sense that multiple speakers are speaking.

I tried one approach to keep adding the embeddings to the existing ark file in the predict.py as below, however the clustering process will be very slow as the .ark file grows -

Can you please suggest any way in which this can be achieved. Kindly let me know for any further information required. Thanks.!!

Part 1, At the start -

ark_dict_existing = {}
ark_dict_exist = False
if os.path.isfile(out_ark_file):
    ark_dict_exist = True
    for key, numpy_array in kaldi_io.read_vec_flt_ark(out_ark_file):
        ark_dict_existing[key] = numpy_array

Part 2 , At the end -

if ark_dict_exist:
    ark_dict_temp = {}
    for key, numpy_array in kaldi_io.read_vec_flt_ark(out_ark_file):
        ark_dict_temp[key] = numpy_array

    ark_dict_new = dict(ark_dict_existing, **ark_dict_temp)

    with open(out_ark_file, 'wb') as f:
        for key, vec in ark_dict_new.items():
            kaldi_io.write_vec_flt(f, vec, key=key)

Regards,
Vinay

More detail for package version

Hello,

I wanna to reproduce this work more accurately.

May I get more details what you used?

Maybe is numpy version or others information?

Update links to new paper

https://authors.elsevier.com/a/1dN0k39HpSeKMh

How to generate *.lab file for custom *.wav file

Hello,

For the predict.py we need "--in-lab-dir example/vad", how can we generate *.lab file for the custom *.wav file.

Can you please provide more details on this file and its generation/availability.

Thanks.

How to process 1h30m of audio?

I'm using the run_example.sh script to get the rttm. It works for most of the files, however the oom_killer kills the process for the following conversation in wav format (ffprobe output):

Duration: 01:21:25.15, bitrate: 256 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1 channels, s16, 256 kb/s

I created a dummy .lab file with a single line 0.000 4885.153375 sp.

How can I process this long file?

Transition matrix update in every iteration

In the VB_diarization() method, it seems that the transition matrix tr gets updated in each iteration at line

VBx/VBx/VB_diarization.py

Line 124 in fbc5a1e

tr[(np.arange(1, maxSpeakers+1) * minDur - 1,) * 2] += loopProb

so the self loop probability keeps increasing? Is this by design?

couldn't get dscore submodule

Hi,
I am following the steps in README. When initializing the submodule dscore, the git submodule commands return nothing in the terminal (like this)

(vbx) liaozty20@a11:~/VBx$ git submodule init
(vbx) liaozty20@a11:~/VBx$ git submodule update
(vbx) liaozty20@a11:~/VBx$ 

and the dscore directory is not created. The provided example works except that it can't find dscore/score.py. I try to see config in .git but I can't find anything about the submodule.

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://github.com/BUTSpeechFIT/VBx.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

where is the 'dscore/score.py ' in the 'run_example'

Dear author:
Thanks for your program, but I don't see the script named 'dscore/score.py ' in the program. Could you give some guidelines for me ?

Clarification about VAD: oracle or automatic?

I'd like a clarification on the numbers reported in the paper.
Do you confirm that those are based on oracle VAD?

VB clustering

I'm research about VB clustering, but i don't understand its idea. How can it cluster speaker after using ahc. Can you summarize its main idea for me? Thank you

use the score.py appear :[winerror 193]%1 is not a valid win32 application is happening

May be it is also the problem of my pc operation system，so i use the pyannote.mertic to compute the DER ,but i found that the DER is 26.28%, the result in readme, the DER on ES2005a.wav is only 7.06%. I don't know wheather my code is right,could someone figure out the error of my code?

`from future import print_function
from future import unicode_literals

from pyannote.core import Segment, Annotation
from pyannote.metrics.diarization import DiarizationErrorRate

from dscore.scorelib.rttm import load_rttm

ref_rttm = './example/rttm/ES2005a.rttm'

hpy_rttm = './exp/ES2005a.rttm'

ref_turns, ref_speaker_ids_set, _ = load_rttm(ref_rttm)
ref_speaker_ids_lsit = list(ref_speaker_ids_set)

reference = Annotation()
for i in range(len(ref_turns)):
onset = ref_turns[i].onset
offset = ref_turns[i].dur + ref_turns[i].onset
spk_id = ref_turns[i].speaker_id
reference[Segment(onset, offset)] = spk_id

hpy_turns, hpy_speaker_ids_set, _ = load_rttm(hpy_rttm)
hpy_speaker_ids_list = list(hpy_speaker_ids_set)
hypothesis = Annotation()
for i in range(len(hpy_turns)):
onset = hpy_turns[i].onset
offset = hpy_turns[i].dur + hpy_turns[i].onset
spk_id = hpy_turns[i].speaker_id
hypothesis[Segment(onset, offset)] = spk_id

metric = DiarizationErrorRate()
metric(reference, hypothesis)
report = metric.report(display=True)`

Is it possible to run clustering algorithm on GPU?

I used this toolkit on some long audio files. The clustering stage takes long time, like a couple of hours for a 1.5 hour audio (full of speech). Is this normal? Is it possible to run the clustering algorithm on GPU? Thanks!

twoGMMcalib_lin's theory

Hello, I want to know why use this method to calculate of the threshold of AHC? Is there any theory based on? Thank you.

Speaker Recognition

Hello,

Can you please help/suggest using the provided code base how the biometrics speaker recognition (speaker names - who spoke when) task can be accomplished.
In general the cosine/euclidean distance is calculated using the saved speaker embeddings. How can you please suggest how we can achieve biometrics speaker recognition with your approach.

Thanks a lot..!!

ERROR: Unable to open RTTM file: exp/ES2005a.rttm

I am facing this error while running run_example script. I have installed all the packages. Please see below the full error,

#############################################################################
###
### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kal
di-trunk
###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profi
le)
###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)
#############################################################################
###
Traceback (most recent call last):
  File "/tmp/VBx/VBx/predict.py", line 15, in <module>
    import soundfile as sf
  File "/home/khanreaper/anaconda3/envs/my_env/lib/python3.9/site-packages/so
undfile.py", line 142, in <module>
    raise OSError('sndfile library not found')
OSError: sndfile library not found
#############################################################################
###
### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kal
di-trunk
###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profi
le)
###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)
#############################################################################
###
Traceback (most recent call last):
  File "/tmp/VBx/VBx/vbhmm.py", line 91, in <module>
    segs_dict = read_xvector_timing_dict(args.segments_file)
  File "/tmp/VBx/VBx/diarization_lib.py", line 137, in read_xvector_timing_di
ct
    segs = np.loadtxt(kaldi_segments, dtype=object)
  File "/home/khanreaper/anaconda3/envs/my_env/lib/python3.9/site-packages/nu
mpy/lib/npyio.py", line 1065, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/home/khanreaper/anaconda3/envs/my_env/lib/python3.9/site-packages/nu
mpy/lib/_datasource.py", line 194, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/home/khanreaper/anaconda3/envs/my_env/lib/python3.9/site-packages/nu
mpy/lib/_datasource.py", line 531, in open
    raise IOError("%s not found." % path)
OSError: exp/ES2005a.seg not found.
Loading speaker turns from reference RTTMs...
Loading speaker turns from system RTTMs...
ERROR: Unable to open RTTM file: exp/ES2005a.rttm

My conda environment is setup as follows,

WARNING: The conda.compat module is deprecated and will be removed in a future release.
# packages in environment at /home/khanreaper/anaconda3/envs/my_env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
ca-certificates           2021.4.13            h06a4308_1  
certifi                   2020.12.5        py39h06a4308_0  
cffi                      1.14.5                   pypi_0    pypi
h5py                      3.2.1                    pypi_0    pypi
intervaltree              3.1.0                    pypi_0    pypi
joblib                    1.0.1                    pypi_0    pypi
kaldi-io                  0.9.4                    pypi_0    pypi
ld_impl_linux-64          2.33.1               h53a641e_7  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
ncurses                   6.2                  he6710b0_1  
numexpr                   2.7.3                    pypi_0    pypi
numpy                     1.20.2                   pypi_0    pypi
onnxruntime               1.7.0                    pypi_0    pypi
openssl                   1.1.1k               h27cfd23_0  
pip                       21.0.1           py39h06a4308_0  
protobuf                  3.15.8                   pypi_0    pypi
pycparser                 2.20                     pypi_0    pypi
python                    3.9.4                hdb3f193_0  
readline                  8.1                  h27cfd23_0  
scikit-learn              0.24.2                   pypi_0    pypi
scipy                     1.6.3                    pypi_0    pypi
setuptools                52.0.0           py39h06a4308_0  
six                       1.15.0                   pypi_0    pypi
sklearn                   0.0                      pypi_0    pypi
sortedcontainers          2.3.0                    pypi_0    pypi
soundfile                 0.10.3.post1             pypi_0    pypi
sqlite                    3.35.4               hdfb4753_0  
tabulate                  0.8.9                    pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
tk                        8.6.10               hbc83047_0  
torch                     1.8.1                    pypi_0    pypi
typing-extensions         3.10.0.0                 pypi_0    pypi
tzdata                    2020f                h52ac0ba_0  
wheel                     0.36.2             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
zlib                      1.2.11               h7b6447c_3  

Can you please help with this?

Attempting to fix maxSpeakers to 2 sometimes yields a forward-backward error

I have a rather long audio file, about 34min, which I have run with run_example.sh, and my own appropriate .lab file for this recording. For most recordings this works great. But for one I have encountered the following issue when fixing the maxSpeakers variable to 2 (it works if I don't specify, but then it identifies 3 speakers, and I know there are only 2). So far I have been unsuccessful going through the logs.

[-57.92396578 -71.68007889   0.           0.           0.
   0.           0.           0.           0.           0.
   0.           0.        ] [-0.69314718 -0.69314718]
Traceback (most recent call last):
  File "VBx/vbhmm.py", line 147, in <module>
    loopProb=args.loopP, Fa=args.Fa, Fb=args.Fb)
  File "VBx/VB_diarization.py", line 130, in VB_diarization
    gamma, tll, lf, lb = forward_backward(lls.repeat(minDur,axis=1), tr, ip) 
    #, np.arange(1,maxSpeakers+1)*minDur-1)
  File "VBx/VB_diarization.py", line 285, in forward_backward
    lfw[0] = lls[0] + np.log(ip)
ValueError: operands could not be broadcast together with shapes (12,) (2,)

Low Accuracy for 15 sec or 30 sec audio clips

Hello,

I have a set of 15 or 30 sec clips, however the model provides less accurate results for these short clips. If the clips are merged into larger audio file then the results improve. I tried changing some of the input parameters but to no success.

Can you please suggest if any of the settings can be changed for the 15 sec or 30 sec audio clips so that the accuracy can be improved for them as well.

Thanks a lot,
Vinay

Can VBx work on the windows?

i am very interested in this programe when i run the shell file on my windows pc . The warning appear below,i really don't know what it's mean.

WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kaldi-trunk

(please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profile)

(or run as: KALDI_ROOT=<your_path> python <your_script>.py)

what are ref.rttm and sys.rttm meaning?

Hello~

In the first, I think that ref.rttm is reference rttm (ground truth) and sys.rttm is diarization system output.

But recently I find that maybe something is wrong....

What are they really meaning?

AHC DER

Hello,

Recently, I am trying to reproduce the AHC's DER 21.43% on paper.

I just change METHOD=AHC+VB to METHOD=AHC.

However, I get AHC DER 28.19%.

By the way, the DER for VBx I got is 19.07%.

I think the 6.76% is beyond the range of random error, but I don't know why...

I used the pretrained model raw_81.pth for x-vector.

I compared the ark file generated by myself with the ark file generated by you.

They are just a bit different.

When I tried ./run_example.sh, I got the same result which you provided.

Do you have any suggestions for this?

.

ARK generated by myself.

[[-0.16536108  0.09801682  0.11645849 ...  0.09223148 -0.15511973
  -0.23234421]
 [-0.24052846  0.10655481 -0.06635439 ...  0.10831092 -0.1426015
  -0.19911067]
 [-0.13864656  0.08062537 -0.14771278 ...  0.06587379 -0.14089754
  -0.19367374]
 ...
 [ 0.01711985  0.03500973  0.0656618  ...  0.07045757 -0.05498593
   0.00638826]
 [ 0.00529743  0.04878493  0.06871074 ...  0.06359828 -0.05343872
  -0.00458009]
 [ 0.00332284  0.02749975  0.0702526  ...  0.0678948  -0.06131358
  -0.00629377]]

ARK generated by myself.

[[-0.16536093  0.09801689  0.11645849 ...  0.09223147 -0.15511961
  -0.23234421]
 [-0.24052843  0.10655481 -0.06635463 ...  0.10831097 -0.14260143
  -0.19911063]
 [-0.1386464   0.08062521 -0.14771265 ...  0.06587409 -0.14089729
  -0.19367379]
 ...
 [ 0.01711991  0.03500969  0.06566183 ...  0.07045754 -0.05498599
   0.00638824]
 [ 0.00529744  0.04878489  0.06871083 ...  0.06359822 -0.05343874
  -0.00458009]
 [ 0.00332281  0.02749969  0.07025264 ...  0.06789468 -0.06131361
  -0.00629374]]

AHC's result full by myself.

File               DER    JER    B3-Precision    B3-Recall    B3-F1    GKT(ref, sys)    GKT(sys, ref)    H(ref|sys)    H(sys|ref)    MI    NMI
---------------  -----  -----  --------------  -----------  -------  ---------------  ---------------  ------------  ------------  ----  -----
EN2002a          22.79  29.88            0.76         0.68     0.72             0.61             0.70          0.65          1.37  1.80   0.65
EN2002b          27.12  38.67            0.65         0.67     0.66             0.53             0.56          0.92          1.35  1.50   0.57
EN2002c          19.12  32.44            0.57         0.74     0.64             0.55             0.42          1.14          1.20  0.95   0.45
EN2002d          26.07  34.24            0.78         0.65     0.71             0.57             0.72          0.67          1.42  1.78   0.64
ES2004a          13.42  16.57            0.96         0.82     0.88             0.77             0.94          0.13          0.73  2.02   0.83
ES2004b           6.20   6.41            0.97         0.90     0.93             0.87             0.96          0.10          0.47  2.14   0.88
ES2004c           8.08   9.32            0.97         0.87     0.92             0.84             0.96          0.11          0.61  2.13   0.86
ES2004d          14.04  18.13            0.94         0.80     0.87             0.76             0.93          0.18          0.89  2.04   0.80
IS1009a          12.18  19.40            0.95         0.82     0.88             0.74             0.92          0.15          0.70  1.61   0.80
IS1009b           7.61   7.60            0.97         0.87     0.92             0.84             0.97          0.08          0.62  2.17   0.87
IS1009c           8.63   9.13            0.98         0.86     0.92             0.83             0.98          0.06          0.61  2.18   0.87
IS1009d          10.63   9.82            0.97         0.84     0.90             0.79             0.96          0.10          0.74  1.93   0.83
TS3003a          25.22  24.02            0.98         0.71     0.83             0.57             0.97          0.04          1.18  1.13   0.69
TS3003b          14.93  11.50            0.92         0.78     0.85             0.73             0.90          0.19          0.95  1.95   0.78
TS3003c          10.37   9.63            0.98         0.85     0.91             0.81             0.98          0.05          0.68  2.18   0.86
TS3003d          17.52  21.60            0.91         0.77     0.84             0.71             0.87          0.27          1.01  1.86   0.75
*** OVERALL ***  15.20  19.24            0.88         0.79     0.83             0.79             0.88          0.34          0.92  5.78   0.90

VBx's result full by myself.

File               DER    JER    B3-Precision    B3-Recall    B3-F1    GKT(ref, sys)    GKT(sys, ref)    H(ref|sys)    H(sys|ref)    MI    NMI
---------------  -----  -----  --------------  -----------  -------  ---------------  ---------------  ------------  ------------  ----  -----
EN2002a           0.20   0.62            1.00         1.00     1.00             1.00             1.00          0.02          0.02  2.43   0.99
EN2002b           0.40   1.33            0.99         0.99     0.99             0.99             0.99          0.03          0.03  2.40   0.99
EN2002c           0.44   1.89            0.99         0.99     0.99             0.99             0.99          0.03          0.04  2.06   0.98
EN2002d           0.17   0.48            1.00         1.00     1.00             1.00             1.00          0.01          0.02  2.44   0.99
ES2004a           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.15   1.00
ES2004b           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.24   1.00
ES2004c           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.24   1.00
ES2004d           0.03   0.10            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.22   1.00
IS1009a           0.41   1.07            0.99         0.99     0.99             0.99             0.99          0.03          0.03  1.73   0.98
IS1009b           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.25   1.00
IS1009c           0.02   0.03            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.24   1.00
IS1009d           0.93   1.08            0.99         0.98     0.99             0.98             0.99          0.03          0.07  2.01   0.98
TS3003a           0.17   1.17            1.00         1.00     1.00             1.00             1.00          0.01          0.01  1.17   0.99
TS3003b           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.14   1.00
TS3003c           0.00   0.00            1.00         1.00     1.00             1.00             1.00          0.00          0.00  2.23   1.00
TS3003d           0.32   3.05            1.00         1.00     1.00             0.99             0.99          0.01          0.02  2.11   0.99
*** OVERALL ***   0.19   0.70            1.00         1.00     1.00             1.00             1.00          0.01          0.02  6.11   1.00

AMI DER result cannot be reproduced

hello, I try to repeat AMI DER result, but when I run AMI_run.sh, I can not get the result which report in the paper "Bayesian HMM clustering of x-vector sequences (VBx)bin speaker diarization: theory, implementation and analysis on standard tasks", can you help me to solve this problem?
I'm also not very clear about the mix-headset and beamformed test，is it only a channel or need some tools to generate it ?

VBx written in C++

Hello, I would like to ask if there is a VBx algorithm written in C + +

what is the difference between plda_psi and plda_tr in line 93 and lines 97 and 98?

In vbhmm.py, what is the difference between plda_psi and plda_tr in line 93 and lines 97 and 98? I see that some lines in plda_tr are multiplied by -1.

Sharing the expected output?

Thanks again for sharing this code.

Any chance you could share the expected output RTTMs for all 4 datasets (and corresponding dev/test splits)?

This would be very useful to check that everything went as expected when running the code.
One can obviously compare the final error rates but having the final RTTMs would make this check even more precise.

AMI beamformed wav

Hello~

Your project is so great.
Now, I wanna to try the beamform experiment.
Is your AMI_beamformed based on Kaldi s5b 's --mic mdm8?

Thank you.

"AHC" from library

"AHC" is missing from diarization_lib.py and its called a couple of times in vbhmm.py.

Pytorch backend problem

Hi, I try to use the pytorch backend. I used the ResNet101_16kHz/nnet/raw_81.pth.zip* file (model file md5sum: 4dbae96de5392a9db41193b1aebdfe24 ), but I got error and the log : "Unexpected key(s) in state_dict: "metric.weight". I'm not sure if I'm using it incorrectly. Does it support the pytorch backend now? Thanks......

plda and transform.h5

Hello,

I would like to reproduce the result of this work. May I know how you created the plda file and transform.h5 file? I saw a comment in resnet.py showing 'For Pre-activation ResNet, see 'preact_resnet.py'.'. Could you please also share that file as well?

Thanks a lot!

Kind Regards

Not detecting exact number of speakers

In 45min it is detecting 5 speakers, in ground truth it have 8-10 speakers. Speakers for which voice activity is less (less than 5sec) are given labels of other.
I have not used rttm file.
Replaced audio & vad file in examples sub directory, to test performance.
Why number of speaker is problem?

run_example not giving full output my wav

Dear Author,
I got the following results for a 55 minutes long wav file with a bit rate of 256kbps. I think it is not giving me the full output. What do I have to change in run_example.py to make it compatible with my wav file?

SPEAKER ES2005a 1 0.000000 15.000000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 15.000000 2.160000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 17.160000 1.440000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 18.600000 6.550000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 25.520000 8.970000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 35.900000 1.550000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 38.330000 2.070000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 40.420000 18.120000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 58.540000 4.320000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 62.860000 4.080000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 66.940000 0.480000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 67.420000 13.440000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 80.860000 0.720000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 81.580000 6.960000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 88.540000 1.920000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 90.460000 5.280000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 95.740000 0.960000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 96.700000 1.200000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 97.900000 0.240000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 98.140000 1.680000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 99.820000 2.880000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 102.700000 4.000000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 106.710000 26.950000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 133.880000 2.380000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 137.190000 3.000000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 140.190000 2.570000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 143.040000 2.760000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 145.800000 3.600000 <NA> <NA> 2 <NA> <NA>
SPEAKER ES2005a 1 149.400000 22.320000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 171.720000 7.690000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 180.430000 1.800000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 182.230000 0.720000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 182.950000 15.440000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 199.710000 0.330000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 201.710000 8.760000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 210.470000 3.740000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 214.860000 0.300000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 215.210000 8.800000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 225.080000 1.080000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 226.160000 0.840000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 228.960000 6.360000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 235.320000 3.360000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 238.680000 2.300000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 241.370000 0.960000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 243.630000 1.080000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 244.710000 3.810000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 250.200000 1.590000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 253.100000 1.550000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 255.080000 0.980000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 266.840000 2.430000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 276.780000 0.840000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 277.620000 2.160000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 279.780000 0.720000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 280.500000 2.690000 <NA> <NA> 31 <NA> <NA>
SPEAKER ES2005a 1 283.870000 2.520000 <NA> <NA> 31 <NA> <NA>
SPEAKER ES2005a 1 286.390000 0.720000 <NA> <NA> 3 <NA> <NA>
SPEAKER ES2005a 1 287.110000 12.480000 <NA> <NA> 14 <NA> <NA>
SPEAKER ES2005a 1 299.590000 2.210000 <NA> <NA> 19 <NA> <NA>
SPEAKER ES2005a 1 302.140000 4.450000 <NA> <NA> 19 <NA> <NA>

How to use GPU?

I wanna run repo with GPU but i don't know how to use it. I try set args.gpu = 0 and args.backend = pytorch but there is an error " "The NVIDIA driver on your system is too old ". My server use cuda 10.1.
Thank you!!!!!

Error Handling for the empty/silence audio files

Hello Federico, Good Evening..!!

Currently if we are processing an empty/silent audio file then the *.lab file will be generated as empty and thus the subsequent files will be empty leading to below error. Can you please suggest how can we handle this error efficiently through the code. Exception handling yes can done but wanted to once understand if we can still process such files efficiently in any better way.
Thanks,
Vinay

Error-
CRITICAL - index 0 is out of bounds for axis 1 with size 0
Traceback (most recent call last):
File “/services/speech_diarization/service_streaming_unsupervised.py", line 169, in get_streaming_unsupervised
predict.predict_embeddings(directory_path_session, combined_audio_file, audio_file_path,
File "/services/speech_diarization/unsupervised_diarization/predict_streaming.py", line 158, in predict_embeddings
seg = signal[labs[segnum, 0]:labs[segnum, 1]]
IndexError: index 0 is out of bounds for axis 1 with size 0
CRITICAL - index 0 is out of bounds for axis 1 with size 0

telephone codec

thank you for this wonderful contribution.

i read the paper,"https://arxiv.org/pdf/2012.14952.pdf"*(F. Landini, J. Profant, M. Diez, L. Burget: Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks).

the 8kHz:Any wide-band data used were
downsampled to 8 kHz and passed through a telephone codec.
i want to know what is the telephone codec.maybe use the g711 to encode and decode?

thank you very much.

Inconsistency on pip installation

Hi all with appreciation to this amazing work!

I had a bit trouble on installing the dependencies via pip: in requirements.txt, there are version inconsistencies between modules, along with pip and python versions. I note problems below:

• There are indicators in the README on the requirement of using python3.7. But actually for numpy 1.22.0, I believe we need later version (I used python3.9)
• For numpy==1.22.0, the scipy==1.4.1 cannot be properly installed.

I actually then tried to install python3.9 and then install all libraries without indicating their version number via pip. I ran ./run_example.sh and perfectly replicated the results in README. Therefore, I think maybe we can update the README. If you think this makes sense but have been too busy, I can do this trivial thing.

Thanks again for open-sourcing this amazing repository and related work to the community!

Division by zero in AHC

Hey, I wanted to ask about a problem in AHC - division by zero.

diarization_lib.py:30: RuntimeWarning: divide by zero encountered in true_divide
  threshold = -0.5 * (np.log(weights ** 2 / var) - means ** 2 / var).dot([1, -1]) / (means / var).dot([1, -1])
diarization_lib.py:30: RuntimeWarning: invalid value encountered in subtract
  threshold = -0.5 * (np.log(weights ** 2 / var) - means ** 2 / var).dot([1, -1]) / (means / var).dot([1, -1])
Traceback (most recent call last):
  File "vbhmm.py", line 130, in <module>
    labels1st = AHC(scr_mx, thr + args.threshold)
  File "diarization_lib.py", line 53, in AHC
    dist[:, mi] = dist[mi, :] = (dist[mi, :] * len(clsts[mi]) + dist[mj, :] * len(clsts[mj])) / (
TypeError: object of type 'NoneType' has no len()

For now, I solved it like this before division by var:

if abs(var) < var_eps:  # avoid division by zero
    break

Create VAD

I don't know how to create vad, i try use auditok to create vad, but with my vad, DER result quite bad,

Error: Empty input file in predict.py

Hi there, I am trying to run the example script, I changed to the 8 kHz model and added a file in a new 8kHz folder where I put one example file. When I run the script, I get this error: Empty input file: "example/vad/9BHU18367S.lab".

I don't get this error when I run the 16 kHz model with the given example file. How do I generate the lab file? My assumption is that if I want to diarise, I just give the input file and model. Thanks.