This function calculates threshold dependent and independent model performance metrics.
Arguments
- p
numeric. Predicted suitability for presences
- a
numeric. Predicted suitability for absences
- bg
numeric. Predicted suitability for background points, used for BOYCE metric. It bg is set as NULL, BOYCE metric will be calculated with presences and absences suitabilities values
- thr
character. Threshold criterion used to get binary suitability values (i.e. 0,1). Used for threshold-dependent performance metrics. It is possible to use more than one threshold type. A vector must be provided for this argument. The following threshold criteria are available:
lpt: The highest threshold at which there is no omission.
equal_sens_spec: Threshold at which the Sensitivity and Specificity are equal.
max_sens_spec: Threshold at which the sum of the Sensitivity and Specificity is the highest (aka threshold that maximizes the TSS).
max_jaccard: The threshold at which the Jaccard index is the highest.
max_sorensen: The threshold at which the Sorensen index is the highest.
max_fpb: The threshold at which FPB (F-measure on presence-background data) is the highest.
sensitivity: Threshold based on a specified Sensitivity value. Usage thr = c('sensitivity', sens='0.6') or thr = c('sensitivity'). 'sens' refers to Sensitivity value. If a sensitivity value is not specified, the default value is 0.9
If more than one threshold type is used, concatenate threshold types, e.g., thr=c('lpt', 'max_sens_spec', 'max_jaccard'), or thr=c('lpt', 'max_sens_spec', 'sensitivity', sens='0.8'), or thr=c('lpt', 'max_sens_spec', 'sensitivity'). Function will use all thresholds if no threshold type is specified
Value
a tibble with next columns
threshold: threshold names
thr_value: threshold values
n_presences: number of presences
n_absences: number of absences
from TPR to IMAE: performance metrics
Details
This function is used for evaluating different models approaches base on the combination of presence-absences or presence-pseudo-absences and background point data and suitability predicted by any model or flexsdm modeling function families (fit_, esm_, and tune_.)
It calculates the next performance metric:
| Performance metric | Threshold dependent | Values ranges |
| TPR (True Positive Rate, also called Sensitivity) | yes | 0 - 1 |
| TNR (True Negative Rate, also called Specificity) | yes | 0 - 1 |
| SORENSEN | yes | 0 - 1 |
| JACCARD | yes | 0 - 1 |
| FPB (F-measure on presence-background) | yes | 0 - 2 |
| OR (Omission Rate) | yes | 0 - 1 |
| TSS (True Skill Statistic) | yes | -1 - 1 |
| KAPPA | yes | 0 - 1 |
| AUC (Area Under Curve) | no | 0 - 1 |
| BOYCE (continuous Boyce index)* | no | -1 - 1 |
| IMAE (Inverse Mean Absolute Error)** | no | 0 - 1 |
\* BOYCE is calculated based on presences and background points, in case that background points is not provided it is calculated using presences and absences. The codes for calculating this metric is and adaptation of enmSdm package (https://github.com/adamlilith/enmSdm)
\** IMAE is calculated as 1-(Mean Absolute Error) in order to be consistent with the other metrics where the higher the value of a given performance metric, the greater the model's accuracy
Examples
if (FALSE) {
require(dplyr)
set.seed(0)
p <- rnorm(50, mean = 0.7, sd = 0.3) %>% abs()
p[p > 1] <- 1
p[p < 0] <- 0
set.seed(0)
a <- rnorm(50, mean = 0.3, sd = 0.2) %>% abs()
a[a > 1] <- 1
a[a < 0] <- 0
set.seed(0)
backg <- rnorm(1000, mean = 0.4, sd = 0.4) %>% abs()
backg[backg > 1] <- 1
backg[backg < 0] <- 0
# Function use without threshold specification
e <- sdm_eval(p, a)
e
# Function use with threshold specification
sdm_eval(p, a, thr = "max_sorensen")
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "max_jaccard"))
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "sensitivity"))
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "sensitivity", sens = "0.95"))
# Use of bg argument (it will only be used for calculating BOYCE index)
sdm_eval(p, a, thr = "max_sens_spec")
sdm_eval(p, a, thr = c("max_sens_spec"), bg = backg)
# If background will be used to calculate all other metrics
# background values can be used in "a" argument
sdm_eval(p, backg, thr = "max_sens_spec")
}