### Example of scoring and summarising Source: https://epiforecasts.io/scoringutils/reference/summarise_scores.html An example showing how to convert sample data to a forecast object and calculate scores. ```R scores <- example_sample_continuous |> as_forecast_sample() |> score() ``` -------------------------------- ### Example of checking a forecast object Source: https://epiforecasts.io/scoringutils/reference/is_forecast.html Demonstrates converting an example to a binary forecast and verifying its class using is_forecast. ```R forecast_binary <- as_forecast_binary(example_binary) #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. is_forecast(forecast_binary) #> [1] TRUE ``` -------------------------------- ### Example: Asserting a Binary Forecast Source: https://epiforecasts.io/scoringutils/reference/assert_forecast.html Demonstrates converting an example binary dataset to a forecast object and validating it with assert_forecast. ```R forecast <- as_forecast_binary(example_binary) #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. assert_forecast(forecast) #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. ``` -------------------------------- ### Example of calculating correlations Source: https://epiforecasts.io/scoringutils/reference/correlation.html An example showing how to calculate correlations for scores generated from example_quantile data. ```R scores <- score(example_quantile) #> The following messages were produced when checking inputs: #> 1. 144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected. correlation(scores) #> Warning: There is a column called 'quantile' in the scores. Usually, you should call 'summarise_scores()' to summarise over quantiles and obtain one score per forecast before calculating correlations. You can ignore this warning if you know what you're doing. #> interval_score dispersion underprediction overprediction coverage #> 1: 1.00 0.41 0.31 0.93 -0.23 #> 2: 0.41 1.00 0.10 0.27 -0.05 #> 3: 0.31 0.10 1.00 -0.03 -0.23 #> 4: 0.93 0.27 -0.03 1.00 -0.16 #> 5: -0.23 -0.05 -0.23 -0.16 1.00 #> 6: -0.23 -0.08 -0.20 -0.17 0.78 #> 7: 0.10 0.10 -0.32 0.21 0.03 #> 8: 0.97 0.47 0.30 0.89 -0.21 #> coverage_deviation bias ae_median metric #> 1: -0.23 0.10 0.97 interval_score #> 2: -0.08 0.10 0.47 dispersion #> 3: -0.20 -0.32 0.30 underprediction #> 4: -0.17 0.21 0.89 overprediction #> 5: 0.78 0.03 -0.21 coverage #> 6: 1.00 0.04 -0.26 coverage_deviation #> 7: 0.04 1.00 0.10 bias #> 8: -0.26 0.10 1.00 ae_median ``` -------------------------------- ### Check Forecasts Example Source: https://epiforecasts.io/scoringutils/reference/check_forecasts.html Example of running check_forecasts on quantile-based forecast data and printing the resulting diagnostic information. ```R check <- check_forecasts(example_quantile) #> The following messages were produced when checking inputs: #> 1. 144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected. print(check) #> Your forecasts seem to be for a target of the following type: #> $target_type #> [1] "integer" #> #> and in the following format: #> $prediction_type #> [1] "quantile" #> #> The unit of a single forecast is defined by: #> $forecast_unit #> [1] "location" "target_end_date" "target_type" "location_name" #> [5] "forecast_date" "model" "horizon" #> #> Cleaned data, rows with NA values in prediction or true_value removed: #> $cleaned_data #> location target_end_date target_type true_value location_name #> 1: DE 2021-05-08 Cases 106987 Germany #> 2: DE 2021-05-08 Cases 106987 Germany #> 3: DE 2021-05-08 Cases 106987 Germany #> 4: DE 2021-05-08 Cases 106987 Germany #> 5: DE 2021-05-08 Cases 106987 Germany #> --- #> 20397: IT 2021-07-24 Deaths 78 Italy #> 20398: IT 2021-07-24 Deaths 78 Italy ``` -------------------------------- ### se_mean_sample example Source: https://epiforecasts.io/scoringutils/reference/se_mean_sample.html Example demonstrating how to generate observed values and predictive samples to calculate the squared error of the mean. ```R observed <- rnorm(30, mean = 1:30) predicted_values <- matrix(rnorm(30, mean = 1:30)) se_mean_sample(observed, predicted_values) #> [1] 4.546623e+00 5.575020e-01 3.935505e+00 9.437016e-01 2.431634e-03 #> [6] 2.094508e-01 3.115281e+00 4.288333e-01 1.944089e+00 1.679750e-04 #> [11] 3.051100e-01 1.809854e-01 1.889211e+00 3.757465e-01 4.818431e-05 #> [16] 1.385961e+00 2.840632e+00 5.637887e-01 3.563894e-01 8.583315e-01 #> [21] 3.606269e+00 1.578801e-01 6.666549e+00 2.816836e+00 5.620241e-01 #> [26] 4.845099e-01 3.809361e+00 3.687691e-01 1.570164e+00 2.714351e+00 ``` -------------------------------- ### Example of selecting a specific metric Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_quantile.html This example demonstrates how to select the 'wis' metric from the available scoring rules for a quantile-based forecast object. ```R get_metrics(example_quantile, select = "wis") #> $wis #> function (observed, predicted, quantile_level, separate_results = FALSE, #> weigh = TRUE, count_median_twice = FALSE, na.rm = FALSE) #> { #> assert_input_quantile(observed, predicted, quantile_level) #> reformatted <- quantile_to_interval(observed, predicted, #> quantile_level) #> interval_ranges <- get_range_from_quantile(quantile_level[quantile_level != #> 0.5]) #> complete_intervals <- duplicated(interval_ranges) | duplicated(interval_ranges, #> fromLast = TRUE) #> if (!all(complete_intervals) && !isTRUE(na.rm)) { #> incomplete <- quantile_level[quantile_level != 0.5][!complete_intervals] #> cli_abort(c(`!` = "Not all quantile levels specified form symmetric prediction\n intervals.\n The following quantile levels miss a corresponding lower/upper bound:\n {.val {incomplete}}.\n You can drop incomplete prediction intervals using `na.rm = TRUE`.")) #> } #> assert_logical(separate_results, len = 1) #> assert_logical(weigh, len = 1) #> assert_logical(count_median_twice, len = 1) #> assert_logical(na.rm, len = 1) #> if (separate_results) { #> cols <- c("wis", "dispersion", "underprediction", "overprediction") #> } #> else { #> cols <- "wis" #> } #> reformatted[, `:=`(eval(cols), do.call(interval_score, list(observed = observed, #> lower = lower, upper = upper, interval_range = interval_range, #> weigh = weigh, separate_results = separate_results)))] #> if (count_median_twice) { #> reformatted[, `:=`(weight, 1)] #> } #> else { #> reformatted[, `:=`(weight, ifelse(interval_range == 0, #> 0.5, 1))] #> } #> reformatted <- reformatted[, lapply(.SD, weighted.mean, na.rm = na.rm, #> w = weight), by = "forecast_id", .SDcols = colnames(reformatted) %like% #> paste(cols, collapse = "|")] #> if (separate_results) { #> return(list(wis = reformatted$wis, dispersion = reformatted$dispersion, #> underprediction = reformatted$underprediction, overprediction = reformatted$overprediction)) #> } #> else { #> return(reformatted$wis) #> } #> } #> #> ``` -------------------------------- ### Example: logs_sample() with simulated data Source: https://epiforecasts.io/scoringutils/reference/logs_sample.html Example showing how to compute log scores for 30 observations using 200 Monte Carlo samples. The output is a vector of scores. ```r observed <- rnorm(30, mean = 1:30) predicted <- replicate(200, rnorm(30, mean = 1:30)) logs_sample(observed, predicted) #> [1] 2.4489536 3.7116350 1.0862273 1.1940239 1.2253578 1.2076263 1.0916030 #> [8] 1.3492135 1.1224192 0.8611935 1.2755833 1.4257878 1.4456369 1.9480844 #> [15] 1.0418232 0.9733666 1.4392295 1.1437437 1.9639265 0.8770563 1.0327465 #> [22] 1.3654438 3.3944156 1.5085371 1.4531986 1.0920996 1.0067286 0.9989672 #> [29] 1.9040067 2.8640134 ``` -------------------------------- ### log_shift examples Source: https://epiforecasts.io/scoringutils/reference/log_shift.html Examples showing basic usage of log_shift with and without an offset, and its application within a forecast transformation pipeline. ```R log_shift(1:10) #> [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 1.7917595 1.9459101 #> [8] 2.0794415 2.1972246 2.3025851 log_shift(0:9, offset = 1) #> [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 1.7917595 1.9459101 #> [8] 2.0794415 2.1972246 2.3025851 example_quantile[observed > 0, ] |> as_forecast_quantile() |> transform_forecasts(fun = log_shift, offset = 1) #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. #> Forecast type: quantile #> Forecast unit: #> location, target_end_date, target_type, location_name, forecast_date, model, #> horizon, and scale #> #> location target_end_date target_type observed location_name #> #> 1: DE 2021-01-02 Cases 1.273000e+05 Germany #> 2: DE 2021-01-02 Deaths 4.534000e+03 Germany #> 3: DE 2021-01-09 Cases 1.549220e+05 Germany #> 4: DE 2021-01-09 Deaths 6.117000e+03 Germany #> 5: DE 2021-01-16 Cases 1.101830e+05 Germany #> --- #> 40672: IT 2021-07-24 Deaths 4.369448e+00 Italy #> 40673: IT 2021-07-24 Deaths 4.369448e+00 Italy #> 40674: IT 2021-07-24 Deaths 4.369448e+00 Italy #> 40675: IT 2021-07-24 Deaths 4.369448e+00 Italy #> 40676: IT 2021-07-24 Deaths 4.369448e+00 Italy #> forecast_date quantile_level predicted model horizon #> #> 1: NA NA NA #> 2: NA NA NA #> 3: NA NA NA #> 4: NA NA NA #> 5: NA NA NA #> --- #> 40672: 2021-07-12 0.850 5.866468 epiforecasts-EpiNow2 2 #> 40673: 2021-07-12 0.900 5.986452 epiforecasts-EpiNow2 2 #> 40674: 2021-07-12 0.950 6.214608 epiforecasts-EpiNow2 2 #> 40675: 2021-07-12 0.975 6.416732 epiforecasts-EpiNow2 2 #> 40676: 2021-07-12 0.990 6.579251 epiforecasts-EpiNow2 2 #> scale #> #> 1: natural #> 2: natural #> 3: natural #> 4: natural #> 5: natural #> --- #> 40672: log #> 40673: log #> 40674: log #> 40675: log #> 40676: log ``` -------------------------------- ### Example of plotting interval coverage Source: https://epiforecasts.io/scoringutils/reference/plot_interval_coverage.html An example showing the workflow of converting forecast data, calculating coverage, and plotting the results. ```R example <- as_forecast_quantile(example_quantile) #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. coverage <- get_coverage(example, by = "model") plot_interval_coverage(coverage) ``` -------------------------------- ### Inspect and set forecast units Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html Displays the column names of the quantile example data and demonstrates how to manually set the forecast unit using set_forecast_unit(). ```R colnames(example_quantile) #> [1] "location" "target_end_date" "target_type" "true_value" #> [5] "location_name" "forecast_date" "quantile" "prediction" #> [9] "model" "horizon" set_forecast_unit( example_quantile, c("location", "target_end_date", "target_type", "horizon", "model") ) %>% colnames() #> [1] "true_value" "quantile" "prediction" "location" #> [5] "target_end_date" "target_type" "horizon" "model" ``` -------------------------------- ### Example: plot_wis with relative contributions Source: https://epiforecasts.io/scoringutils/reference/plot_wis.html Example showing how to compute scores and plot WIS contributions with relative_contributions = TRUE, faceted by target_type. ```r library(ggplot2) scores <- example_quantile |> as_forecast_quantile() |> score() #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. scores <- summarise_scores(scores, by = c("model", "target_type")) plot_wis(scores, x = "model", relative_contributions = TRUE ) + facet_wrap(~target_type) ``` -------------------------------- ### Example: plot_avail_forecasts with facet_wrap Source: https://epiforecasts.io/scoringutils/reference/plot_avail_forecasts.html Example that creates an availability plot from `avail_forecasts()` output, using `target_end_date` on the x-axis and faceting by `target_type`. The `show_numbers` argument is set to `FALSE` to hide counts. ```r library(ggplot2) avail_forecasts <- avail_forecasts( example_quantile, by = c("model", "target_type", "target_end_date") ) #> The following messages were produced when checking inputs: #> 1. 144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected. plot_avail_forecasts( avail_forecasts, x = "target_end_date", show_numbers = FALSE ) + facet_wrap("target_type") ``` -------------------------------- ### Example: Plotting a heatmap of bias Source: https://epiforecasts.io/scoringutils/reference/plot_heatmap.html This example demonstrates preparing quantile forecast scores and visualizing the bias metric across target types using plot_heatmap. ```R scores <- example_quantile |> as_forecast_quantile() |> score() #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. scores <- summarise_scores(scores, by = c("model", "target_type")) scores <- summarise_scores( scores, by = c("model", "target_type"), fun = signif, digits = 2 ) plot_heatmap(scores, x = "target_type", metric = "bias") ``` -------------------------------- ### Validate forecast units with check_forecasts Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html Use set_forecast_unit to define the unique identifier columns for your forecasts and check_forecasts to validate the data structure. This example demonstrates an error triggered when forecasts are not uniquely identified. ```r example_quantile %>% set_forecast_unit( c("location", "target_end_date", "forecast_date", "model", "horizon") ) %>% check_forecasts() #> The following messages were produced when checking inputs: #> 1. 144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected. #> Warning in check_forecasts(.): The following warnings were produced when checking inputs: #> 1. Some forecasts have different numbers of rows #> 2. (e.g. quantiles or samples). #> 3. scoringutils found: #> 4. 46, 23 #> 5. . This may be a problem (it can potentially distort scores, #> 6. making it more difficult to compare them), #> 7. so make sure this is intended. #> Error in check_forecasts(.): The following errors were produced when checking inputs: #> 1. There are instances with more than one forecast for the same target. This can't be right and needs to be resolved. Maybe you need to check the unit of a single forecast and add missing columns? Use the function find_duplicates() to identify duplicate rows. ``` -------------------------------- ### Identify duplicate forecasts with find_duplicates Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html Use find_duplicates to investigate instances where multiple forecasts exist for the same target. This example filters the results to inspect specific median forecasts for the EuroCOVIDhub-ensemble. ```r duplicates <- example_quantile %>% set_forecast_unit( c("location", "target_end_date", "forecast_date", "model", "horizon") ) %>% find_duplicates() duplicates[quantile == 0.5 & model == "EuroCOVIDhub-ensemble", ] %>% head() #> true_value quantile prediction location target_end_date forecast_date #> 1: 106987 0.5 119258 DE 2021-05-08 2021-05-03 #> 2: 1582 0.5 1568 DE 2021-05-08 2021-05-03 #> 3: 64985 0.5 110716 DE 2021-05-15 2021-05-03 #> 4: 64985 0.5 92649 DE 2021-05-15 2021-05-10 #> 5: 1311 0.5 1521 DE 2021-05-15 2021-05-03 #> 6: 1311 0.5 1422 DE 2021-05-15 2021-05-10 #> model horizon #> 1: EuroCOVIDhub-ensemble 1 #> 2: EuroCOVIDhub-ensemble 1 #> 3: EuroCOVIDhub-ensemble 2 #> 4: EuroCOVIDhub-ensemble 1 #> 5: EuroCOVIDhub-ensemble 2 #> 6: EuroCOVIDhub-ensemble 1 ``` -------------------------------- ### Examples of get_metrics with binary forecasts Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_binary.html Demonstrates retrieving all default metrics, selecting specific metrics, and excluding specific metrics from the result. ```R get_metrics(example_binary) #> $brier_score #> function (observed, predicted) #> { #> assert_input_binary(observed, predicted) #> observed <- as.numeric(observed) - 1 #> brierscore <- (observed - predicted)^2 #> return(brierscore) #> } #> #> #> #> $log_score #> function (observed, predicted) #> { #> assert_input_binary(observed, predicted) #> observed <- as.numeric(observed) - 1 #> logs <- -log(1 - abs(observed - predicted)) #> return(logs) #> } #> #> #> get_metrics(example_binary, select = "brier_score") #> $brier_score #> function (observed, predicted) #> { #> assert_input_binary(observed, predicted) #> observed <- as.numeric(observed) - 1 #> brierscore <- (observed - predicted)^2 #> return(brierscore) #> } #> #> #> get_metrics(example_binary, exclude = "log_score") #> $brier_score #> function (observed, predicted) #> { #> assert_input_binary(observed, predicted) #> observed <- as.numeric(observed) - 1 #> brierscore <- (observed - predicted)^2 #> return(brierscore) #> } #> #> #> ``` -------------------------------- ### Install scoringutils from CRAN Source: https://epiforecasts.io/scoringutils/index.html Installs the stable CRAN version of the scoringutils package. ```R install.packages("scoringutils") ``` -------------------------------- ### Example: sample_to_quantile(example_integer) Source: https://epiforecasts.io/scoringutils/reference/sample_to_quantile.html Demonstrates converting the `example_integer` dataset to quantile format. The output shows a data.frame in long interval range format with columns including location, model, horizon, true_value, quantile, and prediction. ```r sample_to_quantile(example_integer) #> location location_name target_end_date target_type forecast_date #> 1: DE Germany 2021-01-02 Cases #> 2: DE Germany 2021-01-02 Cases #> 3: DE Germany 2021-01-02 Cases #> 4: DE Germany 2021-01-02 Cases #> 5: DE Germany 2021-01-02 Cases #> --- #> 5151: IT Italy 2021-07-24 Deaths 2021-07-12 #> 5152: IT Italy 2021-07-24 Deaths 2021-07-12 #> 5153: IT Italy 2021-07-24 Deaths 2021-07-12 #> 5154: IT Italy 2021-07-24 Deaths 2021-07-12 #> 5155: IT Italy 2021-07-24 Deaths 2021-07-12 #> model horizon true_value quantile prediction #> 1: NA 127300 0.05 NA #> 2: NA 127300 0.25 NA #> 3: NA 127300 0.50 NA #> 4: NA 127300 0.75 NA #> 5: NA 127300 0.95 NA #> --- #> 5151: epiforecasts-EpiNow2 2 78 0.05 44.90 #> 5152: epiforecasts-EpiNow2 2 78 0.25 89.25 #> 5153: epiforecasts-EpiNow2 2 78 0.50 151.50 #> 5154: epiforecasts-EpiNow2 2 78 0.75 208.00 #> 5155: epiforecasts-EpiNow2 2 78 0.95 469.10 ``` -------------------------------- ### Example: scoring quantile predictions with interval_score() Source: https://epiforecasts.io/scoringutils/reference/interval_score.html Example demonstrating how to compute interval scores for a vector of observed values and prediction intervals. The example generates random data and calls interval_score() with default settings. ```R observed <- rnorm(30, mean = 1:30) interval_range <- rep(90, 30) alpha <- (100 - interval_range) / 100 lower <- qnorm(alpha / 2, rnorm(30, mean = 1:30)) upper <- qnorm((1 - alpha / 2), rnorm(30, mean = 11:40)) scoringutils:::interval_score( observed = observed, lower = lower, upper = upper, interval_range = interval_range ) #> [1] 0.7984288 0.6188294 0.7680146 0.7392137 2.4496739 0.6725337 0.5966822 #> [8] 0.6659581 0.6581834 1.2866256 0.7423596 0.6155816 0.6176548 0.6484069 #> [15] 0.6584252 0.6929587 0.7558694 0.6724925 0.5716195 0.6251716 0.7431658 #> [22] 0.6906088 0.6524924 0.6274762 1.8011311 0.5840689 0.6602319 0.7076748 #> [29] 0.6060337 0.5491802 ``` -------------------------------- ### Example: compute correlations with get_correlations() Source: https://epiforecasts.io/scoringutils/reference/get_correlations.html Example workflow: score example_quantile data and then call get_correlations() to obtain a correlation matrix. The output shows correlations between metrics like wis, overprediction, underprediction, dispersion, bias, interval_coverage_50, interval_coverage_90, and ae_median. ```r scores <- example_quantile |> as_forecast_quantile() |> score() #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. get_correlations(scores) #> wis overprediction underprediction dispersion bias #> #> 1: 1.0000000 0.94297565 0.28377361 0.45566303 0.10545891 #> 2: 0.9429757 1.00000000 -0.03310356 0.32493799 0.21532161 #> 3: 0.2837736 -0.03310356 1.00000000 0.14580143 -0.35123801 #> 4: 0.4556630 0.32493799 0.14580143 1.00000000 0.11118365 #> 5: 0.1054589 0.21532161 -0.35123801 0.11118365 1.00000000 #> 6: -0.2076649 -0.14556039 -0.21392764 -0.09400664 0.01338140 #> 7: -0.4075613 -0.31824017 -0.35756699 -0.08614678 0.09802725 #> 8: 0.9886108 0.90326672 0.33589892 0.53809741 0.09578751 #> interval_coverage_50 interval_coverage_90 ae_median metric #> #> 1: -0.20766492 -0.40756133 0.98861080 wis #> 2: -0.14556039 -0.31824017 0.90326672 overprediction #> 3: -0.21392764 -0.35756699 0.33589892 underprediction #> 4: -0.09400664 -0.08614678 0.53809741 dispersion #> 5: 0.01338140 0.09802725 0.09578751 bias #> 6: 1.00000000 0.37245118 -0.24559356 interval_coverage_50 #> 7: 0.37245118 1.00000000 -0.41079097 interval_coverage_90 #> 8: -0.24559356 -0.41079097 1.00000000 ae_median ``` -------------------------------- ### Install development version from GitHub Source: https://epiforecasts.io/scoringutils/index.html Installs the unstable development version of scoringutils from GitHub with dependencies. ```R remotes::install_github("epiforecasts/scoringutils", dependencies = TRUE) ``` -------------------------------- ### set_forecast_unit() example with example_quantile Source: https://epiforecasts.io/scoringutils/reference/set_forecast_unit.html Example call to `set_forecast_unit()` using the `example_quantile` dataset, specifying the forecast unit columns. The output shows the resulting data.table with the forecast unit and key columns. ```r example_quantile |> scoringutils:::set_forecast_unit( c("location", "target_end_date", "target_type", "horizon", "model") ) ``` ```output #> Forecast type: quantile #> Forecast unit: #> location, target_end_date, target_type, horizon, and model #> #> Key: #> observed quantile_level predicted location target_end_date target_type #> #> 1: 127300 NA NA DE 2021-01-02 Cases #> 2: 4534 NA NA DE 2021-01-02 Deaths #> 3: 154922 NA NA DE 2021-01-09 Cases #> 4: 6117 NA NA DE 2021-01-09 Deaths #> 5: 110183 NA NA DE 2021-01-16 Cases #> --- #> 20541: 78 0.850 352 IT 2021-07-24 Deaths #> 20542: 78 0.900 397 IT 2021-07-24 Deaths #> 20543: 78 0.950 499 IT 2021-07-24 Deaths #> 20544: 78 0.975 611 IT 2021-07-24 Deaths #> 20545: 78 0.990 719 IT 2021-07-24 Deaths #> horizon model #> #> 1: NA #> 2: NA #> 3: NA #> 4: NA #> 5: NA #> --- #> 20541: 2 epiforecasts-EpiNow2 #> 20542: 2 epiforecasts-EpiNow2 #> 20543: 2 epiforecasts-EpiNow2 #> 20544: 2 epiforecasts-EpiNow2 #> 20545: 2 epiforecasts-EpiNow2 ``` -------------------------------- ### Example of get_metrics with ordinal forecast Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_ordinal.html Retrieving the default scoring rules for an example ordinal forecast object. ```R get_metrics(example_ordinal) #> $log_score #> function (observed, predicted, predicted_label) #> { #> assert_input_categorical(observed, predicted, predicted_label) #> n <- length(observed) #> if (n == 1) { #> predicted <- matrix(predicted, nrow = 1) #> } #> observed_indices <- as.numeric(observed) #> pred_for_observed <- predicted[cbind(1:n, observed_indices)] #> logs <- -log(pred_for_observed) #> return(logs) #> } #> #> #> #> $rps #> function (observed, predicted, predicted_label) #> { #> assert_input_ordinal(observed, predicted, predicted_label) #> n <- length(observed) #> if (n == 1) { #> predicted <- matrix(predicted, nrow = 1) #> } #> correct_order <- as.numeric(predicted_label) #> ordered_predicted <- predicted[, correct_order] #> rps <- rps_probs(as.numeric(observed), ordered_predicted) #> return(rps) #> } #> #> #> ``` -------------------------------- ### Compute coverage by model Source: https://epiforecasts.io/scoringutils/reference/get_coverage.html This example demonstrates how to convert a quantile forecast object and compute coverage metrics grouped by model. ```R example_quantile |> as_forecast_quantile() |> get_coverage(by = "model") #> ℹ Some rows containing NA values may be removed. This is fine if not #> unexpected. #> model quantile_level interval_range interval_coverage #> #> 1: EuroCOVIDhub-baseline 0.500 0 0.000000000 #> 2: EuroCOVIDhub-baseline 0.450 10 0.085937500 #> 3: EuroCOVIDhub-baseline 0.550 10 0.085937500 #> 4: EuroCOVIDhub-baseline 0.400 20 0.191406250 #> 5: EuroCOVIDhub-baseline 0.600 20 0.191406250 #> 6: EuroCOVIDhub-baseline 0.350 30 0.289062500 #> 7: EuroCOVIDhub-baseline 0.650 30 0.289062500 #> 8: EuroCOVIDhub-baseline 0.300 40 0.375000000 #> 9: EuroCOVIDhub-baseline 0.700 40 0.375000000 #> 10: EuroCOVIDhub-baseline 0.250 50 0.496093750 #> 11: EuroCOVIDhub-baseline 0.750 50 0.496093750 #> 12: EuroCOVIDhub-baseline 0.200 60 0.628906250 #> 13: EuroCOVIDhub-baseline 0.800 60 0.628906250 #> 14: EuroCOVIDhub-baseline 0.150 70 0.773437500 #> 15: EuroCOVIDhub-baseline 0.850 70 0.773437500 #> 16: EuroCOVIDhub-baseline 0.100 80 0.843750000 #> 17: EuroCOVIDhub-baseline 0.900 80 0.843750000 #> 18: EuroCOVIDhub-baseline 0.050 90 0.910156250 #> 19: EuroCOVIDhub-baseline 0.950 90 0.910156250 #> 20: EuroCOVIDhub-baseline 0.025 95 0.925781250 #> 21: EuroCOVIDhub-baseline 0.975 95 0.925781250 #> 22: EuroCOVIDhub-baseline 0.010 98 0.933593750 #> 23: EuroCOVIDhub-baseline 0.990 98 0.933593750 #> 24: EuroCOVIDhub-ensemble 0.500 0 0.003906250 #> 25: EuroCOVIDhub-ensemble 0.450 10 0.148437500 #> 26: EuroCOVIDhub-ensemble 0.550 10 0.148437500 #> 27: EuroCOVIDhub-ensemble 0.400 20 0.250000000 #> 28: EuroCOVIDhub-ensemble 0.600 20 0.250000000 #> 29: EuroCOVIDhub-ensemble 0.350 30 0.386718750 #> 30: EuroCOVIDhub-ensemble 0.650 30 0.386718750 #> 31: EuroCOVIDhub-ensemble 0.300 40 0.519531250 #> 32: EuroCOVIDhub-ensemble 0.700 40 0.519531250 #> 33: EuroCOVIDhub-ensemble 0.250 50 0.632812500 #> 34: EuroCOVIDhub-ensemble 0.750 50 0.632812500 #> 35: EuroCOVIDhub-ensemble 0.200 60 0.667968750 #> 36: EuroCOVIDhub-ensemble 0.800 60 0.667968750 #> 37: EuroCOVIDhub-ensemble 0.150 70 0.753906250 #> 38: EuroCOVIDhub-ensemble 0.850 70 0.753906250 #> 39: EuroCOVIDhub-ensemble 0.100 80 0.816406250 #> 40: EuroCOVIDhub-ensemble 0.900 80 0.816406250 #> 41: EuroCOVIDhub-ensemble 0.050 90 0.902343750 #> 42: EuroCOVIDhub-ensemble 0.950 90 0.902343750 #> 43: EuroCOVIDhub-ensemble 0.025 95 0.941406250 #> 44: EuroCOVIDhub-ensemble 0.975 95 0.941406250 #> 45: EuroCOVIDhub-ensemble 0.010 98 0.968750000 #> 46: EuroCOVIDhub-ensemble 0.990 98 0.968750000 #> 47: epiforecasts-EpiNow2 0.500 0 0.004048583 #> 48: epiforecasts-EpiNow2 0.450 10 0.093117409 #> 49: epiforecasts-EpiNow2 0.550 10 0.093117409 #> 50: epiforecasts-EpiNow2 0.400 20 0.165991903 #> 51: epiforecasts-EpiNow2 0.600 20 0.165991903 #> 52: epiforecasts-EpiNow2 0.350 30 0.230769231 #> 53: epiforecasts-EpiNow2 0.650 30 0.230769231 ```