pyannote.metrics

a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems

An overview of pyannote.metrics is available as an InterSpeech 2017 paper: it is recommended to read it first, to quickly get an idea whether this tool is for you.

Installation

bash $ pip install pyannote.metrics

Documentation

The documentation is available at http://pyannote.github.io/pyannote-metrics.

Sample notebooks are available here.

Citation

If you use pyannote.metrics in your research, please use the following citation:

bibtex @inproceedings{pyannote.metrics, author = {Herv\'e Bredin}, title = {{pyannote.metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems}}, booktitle = {{Interspeech 2017, 18th Annual Conference of the International Speech Communication Association}}, year = {2017}, month = {August}, address = {Stockholm, Sweden}, url = {http://pyannote.github.io/pyannote-metrics}, }

Issues

hello , how to import some metrics with VAD? such as precesion,recall,FPR?

opened on 2022-08-31 09:52:10 by JJun-Guo

Description

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

How to calculate the relavent error rate in diarization overlap?

opened on 2022-08-20 03:12:57 by axuan731

I have a ref rttm file and a sys rttm file, I want to compute the FA = false alarm rate (%) / Miss. = missed detection rate (%) / F1 = F1-score (%) in the paper "End-to-end speaker segmentation for overlap-aware resegmentation".But I cant find the place to evaluate them in pyannote2.0 Thanks for helping!

ImportError: cannot import name 'Pipeline' from 'pyannote.audio'

opened on 2022-04-05 15:48:56 by walidbou6

tried to run the example of speaker segmentation got the following error:

ImportError: cannot import name 'Pipeline' from 'pyannote.audio'

bash pyannote.audio @ https://github.com/pyannote/pyannote-audio/archive/develop.zip pyannote.core==4.4 pyannote.database==4.1.3 pyannote.metrics==3.2 pyannote.pipeline==2.2

WIP : Add gamma

opened on 2022-02-12 08:32:52 by hadware

This is a WIP to add the gamma-agreement as a new potential evaluation metric. It's based on branch from the sister #37 PR

Segmentation metrics fail with empty annotations

opened on 2019-11-25 08:59:36 by hbredin

Description

Segmentation metrics fail with empty annotations.

Steps/Code to Reproduce

```python from pyannote.core import Annotation, Segment reference = Annotation() hypothesis = Annotation()

from pyannote.metrics.diarization import SegmentationPurity purity = SegmentationPurity() purity(reference, hypothesis) traceback ZeroDivisionError: float division by zero ```

python from pyannote.metrics.diarization import SegmentationCoverage coverage = SegmentationCoverage() coverage(reference, hypothesis) traceback ValueError: zero-size array to reduction operation maximum which has no identity

python from pyannote.metrics.diarization import SegmentationPurityCoverageFMeasure both = SegmentationPurityCoverageFMeasure() both(reference, hypothesis) traceback ValueError: zero-size array to reduction operation maximum which has no identity

Expected Results

It does not really make sense to compute segmentation purity and coverage on empty files but I guess a more explicite message should be raised.

Versions

pyannote.core==3.0 pyannote.database==2.3.1 pyannote.metrics==2.1

Purity and coverage should take non-speech regions into account

opened on 2018-11-21 15:49:24 by hbredin

As of now, returning an empty hypothesis will always give perfect purity and coverage. This shouldn't be the case.