Skip to contents

Evaluate the performance of a sound event detection procedure

Source: R/diagnose_detection.R
diagnose_detection.Rd

diagnose_detection evaluates the performance of a sound event detection procedure comparing the output selection table to a reference selection table

Usage

diagnose_detection(reference, detection, by.sound.file = FALSE,
time.diagnostics = FALSE, cores = 1, pb = TRUE, path = NULL, by = NULL,
 macro.average = FALSE, min.overlap = 0.5)

Arguments

reference

Data frame or 'selection.table' (following the warbleR package format) with the reference selections (start and end of the sound events) that will be used to evaluate the performance of the detection, represented by those selections in 'detection'. Must contained at least the following columns: "sound.files", "selec", "start" and "end". It must contain the reference selections that will be used for detection optimization.

detection

Data frame or 'selection.table' with the detections (start and end of the sound events) that will be compared against the 'reference' selections. Must contained at least the following columns: "sound.files", "selec", "start" and "end". It can contain data for additional sound files not found in 'references'. In this case the routine assumes that no sound events are found in those files, so detection from those files are all false positives.

by.sound.file

Logical argument to control whether performance diagnostics are summarized across sound files (when by.sound.file = FALSE, when more than 1 sound file is included in 'reference') or shown separated by sound file. Default is FALSE.

time.diagnostics

Logical argument to control if diagnostics related to the duration of the sound events ("mean.duration.true.positives", "mean.duration.false.positives", "mean.duration.false.negatives" and "proportional.duration.true.positives") are returned (if TRUE). Default is FALSE.

cores

Numeric. Controls whether parallel computing is applied. It specifies the number of cores to be used. Default is 1 (i.e. no parallel computing).

pb

Logical argument to control progress bar. Default is TRUE.

path

Character string containing the directory path where the sound files are located. If supplied then duty cycle (fraction of a sound file in which sounds were detected)is also returned. This feature is more helpful for tuning an energy-based detection. Default is NULL.

by

Character vector with the name of a column in 'reference' for splitting diagnostics. Diagnostics will be returned separated for each level in 'by'. Default is NULL.

macro.average

Logical argument to control if diagnostics are first calculated for each sound file and then averaged across sound files, which can minimize the effect of unbalanced sample sizes between sound files. If FALSE (default) diagnostics are based on aggregated statistics irrespective of sound files. The following indices can be estimated by macro-averaging: overlap, mean.duration.true.positives, mean.duration.false.positives, mean.duration.false.positives, mean.duration.false.negatives, proportional.duration.true.positives, recall and precision (f.score is always derived from recall and precision). Note that when applying macro-averaging, recall and precision are not derived from the true positive, false positive and false negative values returned by the function.

min.overlap

Numeric. Controls the minimum amount of overlap required for a detection and a reference sound for it to be counted as true positive. Default is 0.5. Overlap is measured as intersection over union.

Value

A data frame including the following detection performance diagnostics:

  • detections: total number of detections

  • true.positives: number of sound events in 'reference' that correspond to any detection. Matching is defined as some degree of overlap in time. In a perfect detection routine it should be equal to the number of rows in 'reference'.

  • false.positives: number of detections that don't match (i.e. don't overlap with) any of the sound events in 'reference'. In a perfect detection routine it should be 0.

  • false.negatives: number of sound events in 'reference' that were not detected (not found in 'detection'. In a perfect detection routine it should be 0.

  • splits: number of detections overlapping reference sounds that also overlap with other detections. In a perfect detection routine it should be 0.

  • merges: number of detections that overlap with two or more reference sounds. In a perfect detection routine it should be 0.

  • mean.duration.true.positives: mean duration of true positives (in ms). Only included when time.diagnostics = TRUE.

  • mean.duration.false.positives: mean duration of false positives (in ms). Only included when time.diagnostics = TRUE.

  • mean.duration.false.negatives: mean duration of false negatives (in ms). Only included when time.diagnostics = TRUE.

  • overlap: mean intersection over union overlap of true positives.

  • proportional.duration.true.positives: ratio of duration of true positives to the duration of sound events in 'reference'. In a perfect detection routine it should be 1. Based only on true positives that were not split or merged.

  • duty.cycle: proportion of a sound file in which sounds were detected. Only included when time.diagnostics = TRUE and path is supplied. Useful when conducting energy-based detection as a perfect detection can be obtained with a very low amplitude threshold, which will detect everything, but will produce a duty cycle close to 1.

  • recall: Proportion of sound events in 'reference' that were detected. In a perfect detection routine it should be 1.

  • precision: Proportion of detections that correspond to sound events in 'reference'. In a perfect detection routine it should be 1.

  • f.score: Combines recall and precision as the harmonic mean of these two. Provides a single value for evaluating performance. In a perfect detection routine it should be 1.

Details

The function evaluates the performance of a sound event detection procedure by comparing its output selection table to a reference selection table in which all sound events of interest have been selected. The function takes any overlap between detected sound events and target sound events as true positives. Note that all sound files located in the supplied 'path' will be analyzed even if not all of them are listed in 'reference'. When several possible matching pairs of sound event and detections are found, the optimal set of matching pairs is found through maximum bipartite matching (using the R package igraph). Priority for assigning a detection to a reference is given by the amount of time overlap. 'splits' and 'merge.positives' are also counted (i.e. counted twice) as 'true.positives'. Therefore "true.positives + false.positives = detections".

References

Araya-Salas, M., Smith-Vidaurre, G., Chaverri, G., Brenes, J. C., Chirino, F., Elizondo-Calvo, J., & Rico-Guevara, A. 2022. ohun: an R package for diagnosing and optimizing automatic sound event detection. BioRxiv, 2022.12.13.520253. https://doi.org/10.1101/2022.12.13.520253

See also

optimize_energy_detector, optimize_template_detector

Author

Marcelo Araya-Salas marcelo.araya@ucr.ac.cr)

Examples

{
  # load data
  data("lbh_reference")

  # perfect detection
  diagnose_detection(reference = lbh_reference, detection = lbh_reference)

  # missing one in detection
  diagnose_detection(reference = lbh_reference, detection = lbh_reference[-1, ])

  # an extra one in detection
  diagnose_detection(reference = lbh_reference[-1, ], detection = lbh_reference)

  # with time diagnostics
  diagnose_detection(
    reference = lbh_reference[-1, ],
    detection = lbh_reference, time.diagnostics = TRUE
  )

  # and extra sound file in reference
  diagnose_detection(
    reference = lbh_reference,
    detection =
      lbh_reference[lbh_reference$sound.files != "lbh1", ]
  )

  # and extra sound file in detection
  diagnose_detection(
    reference =
      lbh_reference[lbh_reference$sound.files != "lbh1", ],
    detection = lbh_reference
  )

  # and extra sound file in detection by sound file
  dd <- diagnose_detection(
    reference =
      lbh_reference[lbh_reference$sound.files != "lbh1", ],
    detection = lbh_reference, time.diagnostics = TRUE, by.sound.file = TRUE
  )

  # get summary
  summarize_diagnostic(dd)
}
#>   detections true.positives false.positives false.negatives splits merges
#> 1         19             19               0               0      0      0
#>   overlap recall precision f.score
#> 1       1      1         1       1