scoringutils

https://epiforecasts.io/scoringutils
Admin
scoringutils is an R package that facilitates the evaluation and scoring of probabilistic and point...
Tokens:190,124
Snippets:1,091
Trust Score:9.2
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example of scoring and summarising

Source: https://epiforecasts.io/scoringutils/reference/summarise_scores.html

An example showing how to convert sample data to a forecast object and calculate scores.

```R
scores <- example_sample_continuous |>
 as_forecast_sample() |>
 score()
```

--------------------------------

### Example of checking a forecast object

Source: https://epiforecasts.io/scoringutils/reference/is_forecast.html

Demonstrates converting an example to a binary forecast and verifying its class using is_forecast.

```R
forecast_binary <- as_forecast_binary(example_binary)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
is_forecast(forecast_binary)
#> [1] TRUE
```

--------------------------------

### Example: Asserting a Binary Forecast

Source: https://epiforecasts.io/scoringutils/reference/assert_forecast.html

Demonstrates converting an example binary dataset to a forecast object and validating it with assert_forecast.

```R
forecast <- as_forecast_binary(example_binary)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
assert_forecast(forecast)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
```

--------------------------------

### Example of calculating correlations

Source: https://epiforecasts.io/scoringutils/reference/correlation.html

An example showing how to calculate correlations for scores generated from example_quantile data.

```R
scores <- score(example_quantile)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
correlation(scores)
#> Warning: There is a column called 'quantile' in the scores. Usually, you should call 'summarise_scores()' to summarise over quantiles and obtain one score per forecast before calculating correlations. You can ignore this warning if you know what you're doing.
#>    interval_score dispersion underprediction overprediction coverage
#> 1:           1.00       0.41            0.31           0.93    -0.23
#> 2:           0.41       1.00            0.10           0.27    -0.05
#> 3:           0.31       0.10            1.00          -0.03    -0.23
#> 4:           0.93       0.27           -0.03           1.00    -0.16
#> 5:          -0.23      -0.05           -0.23          -0.16     1.00
#> 6:          -0.23      -0.08           -0.20          -0.17     0.78
#> 7:           0.10       0.10           -0.32           0.21     0.03
#> 8:           0.97       0.47            0.30           0.89    -0.21
#>    coverage_deviation  bias ae_median             metric
#> 1:              -0.23  0.10      0.97     interval_score
#> 2:              -0.08  0.10      0.47         dispersion
#> 3:              -0.20 -0.32      0.30    underprediction
#> 4:              -0.17  0.21      0.89     overprediction
#> 5:               0.78  0.03     -0.21           coverage
#> 6:               1.00  0.04     -0.26 coverage_deviation
#> 7:               0.04  1.00      0.10               bias
#> 8:              -0.26  0.10      1.00          ae_median
```

--------------------------------

### Check Forecasts Example

Source: https://epiforecasts.io/scoringutils/reference/check_forecasts.html

Example of running check_forecasts on quantile-based forecast data and printing the resulting diagnostic information.

```R
check <- check_forecasts(example_quantile)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
print(check)
#> Your forecasts seem to be for a target of the following type:
#> $target_type
#> [1] "integer"
#> 
#> and in the following format:
#> $prediction_type
#> [1] "quantile"
#> 
#> The unit of a single forecast is defined by:
#> $forecast_unit
#> [1] "location"        "target_end_date" "target_type"     "location_name"  
#> [5] "forecast_date"   "model"           "horizon"        
#> 
#> Cleaned data, rows with NA values in prediction or true_value removed:
#> $cleaned_data
#>        location target_end_date target_type true_value location_name
#>     1:       DE      2021-05-08       Cases     106987       Germany
#>     2:       DE      2021-05-08       Cases     106987       Germany
#>     3:       DE      2021-05-08       Cases     106987       Germany
#>     4:       DE      2021-05-08       Cases     106987       Germany
#>     5:       DE      2021-05-08       Cases     106987       Germany
#>    ---                                                              
#> 20397:       IT      2021-07-24      Deaths         78         Italy
#> 20398:       IT      2021-07-24      Deaths         78         Italy
```

--------------------------------

### se_mean_sample example

Source: https://epiforecasts.io/scoringutils/reference/se_mean_sample.html

Example demonstrating how to generate observed values and predictive samples to calculate the squared error of the mean.

```R
observed <- rnorm(30, mean = 1:30)
predicted_values <- matrix(rnorm(30, mean = 1:30))
se_mean_sample(observed, predicted_values)
#>  [1] 4.546623e+00 5.575020e-01 3.935505e+00 9.437016e-01 2.431634e-03
#>  [6] 2.094508e-01 3.115281e+00 4.288333e-01 1.944089e+00 1.679750e-04
#> [11] 3.051100e-01 1.809854e-01 1.889211e+00 3.757465e-01 4.818431e-05
#> [16] 1.385961e+00 2.840632e+00 5.637887e-01 3.563894e-01 8.583315e-01
#> [21] 3.606269e+00 1.578801e-01 6.666549e+00 2.816836e+00 5.620241e-01
#> [26] 4.845099e-01 3.809361e+00 3.687691e-01 1.570164e+00 2.714351e+00
```

--------------------------------

### Example of selecting a specific metric

Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_quantile.html

This example demonstrates how to select the 'wis' metric from the available scoring rules for a quantile-based forecast object.

```R
get_metrics(example_quantile, select = "wis")
#> $wis
#> function (observed, predicted, quantile_level, separate_results = FALSE, 
#>     weigh = TRUE, count_median_twice = FALSE, na.rm = FALSE) 
#> {
#>     assert_input_quantile(observed, predicted, quantile_level)
#>     reformatted <- quantile_to_interval(observed, predicted, 
#>         quantile_level)
#>     interval_ranges <- get_range_from_quantile(quantile_level[quantile_level != 
#>         0.5])
#>     complete_intervals <- duplicated(interval_ranges) | duplicated(interval_ranges, 
#>         fromLast = TRUE)
#>     if (!all(complete_intervals) && !isTRUE(na.rm)) {
#>         incomplete <- quantile_level[quantile_level != 0.5][!complete_intervals]
#>         cli_abort(c(`!` = "Not all quantile levels specified form symmetric prediction\n        intervals.\n        The following quantile levels miss a corresponding lower/upper bound:\n        {.val {incomplete}}.\n        You can drop incomplete prediction intervals using `na.rm = TRUE`."))
#>     }
#>     assert_logical(separate_results, len = 1)
#>     assert_logical(weigh, len = 1)
#>     assert_logical(count_median_twice, len = 1)
#>     assert_logical(na.rm, len = 1)
#>     if (separate_results) {
#>         cols <- c("wis", "dispersion", "underprediction", "overprediction")
#>     }
#>     else {
#>         cols <- "wis"
#>     }
#>     reformatted[, `:=`(eval(cols), do.call(interval_score, list(observed = observed, 
#>         lower = lower, upper = upper, interval_range = interval_range, 
#>         weigh = weigh, separate_results = separate_results)))]
#>     if (count_median_twice) {
#>         reformatted[, `:=`(weight, 1)]
#>     }
#>     else {
#>         reformatted[, `:=`(weight, ifelse(interval_range == 0, 
#>             0.5, 1))]
#>     }
#>     reformatted <- reformatted[, lapply(.SD, weighted.mean, na.rm = na.rm, 
#>         w = weight), by = "forecast_id", .SDcols = colnames(reformatted) %like% 
#>         paste(cols, collapse = "|")]
#>     if (separate_results) {
#>         return(list(wis = reformatted$wis, dispersion = reformatted$dispersion, 
#>             underprediction = reformatted$underprediction, overprediction = reformatted$overprediction))
#>     }
#>     else {
#>         return(reformatted$wis)
#>     }
#> }
#> <bytecode: 0x559a4ec5dc28>
#> <environment: namespace:scoringutils>
```

--------------------------------

### Example: logs_sample() with simulated data

Source: https://epiforecasts.io/scoringutils/reference/logs_sample.html

Example showing how to compute log scores for 30 observations using 200 Monte Carlo samples. The output is a vector of scores.

```r
observed <- rnorm(30, mean = 1:30)
predicted <- replicate(200, rnorm(30, mean = 1:30))
logs_sample(observed, predicted)
#>  [1] 2.4489536 3.7116350 1.0862273 1.1940239 1.2253578 1.2076263 1.0916030
#>  [8] 1.3492135 1.1224192 0.8611935 1.2755833 1.4257878 1.4456369 1.9480844
#> [15] 1.0418232 0.9733666 1.4392295 1.1437437 1.9639265 0.8770563 1.0327465
#> [22] 1.3654438 3.3944156 1.5085371 1.4531986 1.0920996 1.0067286 0.9989672
#> [29] 1.9040067 2.8640134
```

--------------------------------

### log_shift examples

Source: https://epiforecasts.io/scoringutils/reference/log_shift.html

Examples showing basic usage of log_shift with and without an offset, and its application within a forecast transformation pipeline.

```R
log_shift(1:10)
#>  [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 1.7917595 1.9459101
#>  [8] 2.0794415 2.1972246 2.3025851
log_shift(0:9, offset = 1)
#>  [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 1.7917595 1.9459101
#>  [8] 2.0794415 2.1972246 2.3025851

example_quantile[observed > 0, ] |>
  as_forecast_quantile() |>
  transform_forecasts(fun = log_shift, offset = 1)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
#> Forecast type: quantile
#> Forecast unit:
#> location, target_end_date, target_type, location_name, forecast_date, model,
#> horizon, and scale
#> 
#>        location target_end_date target_type     observed location_name
#>          <char>          <Date>      <char>        <num>        <char>
#>     1:       DE      2021-01-02       Cases 1.273000e+05       Germany
#>     2:       DE      2021-01-02      Deaths 4.534000e+03       Germany
#>     3:       DE      2021-01-09       Cases 1.549220e+05       Germany
#>     4:       DE      2021-01-09      Deaths 6.117000e+03       Germany
#>     5:       DE      2021-01-16       Cases 1.101830e+05       Germany
#>    ---                                                                
#> 40672:       IT      2021-07-24      Deaths 4.369448e+00         Italy
#> 40673:       IT      2021-07-24      Deaths 4.369448e+00         Italy
#> 40674:       IT      2021-07-24      Deaths 4.369448e+00         Italy
#> 40675:       IT      2021-07-24      Deaths 4.369448e+00         Italy
#> 40676:       IT      2021-07-24      Deaths 4.369448e+00         Italy
#>        forecast_date quantile_level predicted                model horizon
#>               <Date>          <num>     <num>               <char>   <num>
#>     1:          <NA>             NA        NA                 <NA>      NA
#>     2:          <NA>             NA        NA                 <NA>      NA
#>     3:          <NA>             NA        NA                 <NA>      NA
#>     4:          <NA>             NA        NA                 <NA>      NA
#>     5:          <NA>             NA        NA                 <NA>      NA
#>    ---                                                                    
#> 40672:    2021-07-12          0.850  5.866468 epiforecasts-EpiNow2       2
#> 40673:    2021-07-12          0.900  5.986452 epiforecasts-EpiNow2       2
#> 40674:    2021-07-12          0.950  6.214608 epiforecasts-EpiNow2       2
#> 40675:    2021-07-12          0.975  6.416732 epiforecasts-EpiNow2       2
#> 40676:    2021-07-12          0.990  6.579251 epiforecasts-EpiNow2       2
#>          scale
#>         <char>
#>     1: natural
#>     2: natural
#>     3: natural
#>     4: natural
#>     5: natural
#>    ---        
#> 40672:     log
#> 40673:     log
#> 40674:     log
#> 40675:     log
#> 40676:     log
```

--------------------------------

### Example of plotting interval coverage

Source: https://epiforecasts.io/scoringutils/reference/plot_interval_coverage.html

An example showing the workflow of converting forecast data, calculating coverage, and plotting the results.

```R
example <- as_forecast_quantile(example_quantile)
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
coverage <- get_coverage(example, by = "model")
plot_interval_coverage(coverage)
```

--------------------------------

### Inspect and set forecast units

Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html

Displays the column names of the quantile example data and demonstrates how to manually set the forecast unit using set_forecast_unit().

```R
colnames(example_quantile)
#>  [1] "location"        "target_end_date" "target_type"     "true_value"     
#>  [5] "location_name"   "forecast_date"   "quantile"        "prediction"     
#>  [9] "model"           "horizon"

set_forecast_unit(
  example_quantile,
  c("location", "target_end_date", "target_type", "horizon", "model")
) %>%
  colnames()
#> [1] "true_value"      "quantile"        "prediction"      "location"       
#> [5] "target_end_date" "target_type"     "horizon"         "model"
```

--------------------------------

### Example: plot_wis with relative contributions

Source: https://epiforecasts.io/scoringutils/reference/plot_wis.html

Example showing how to compute scores and plot WIS contributions with relative_contributions = TRUE, faceted by target_type.

```r
library(ggplot2)
scores <- example_quantile |>
  as_forecast_quantile() |>
  score()
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
scores <- summarise_scores(scores, by = c("model", "target_type"))

plot_wis(scores,
  x = "model",
  relative_contributions = TRUE
) +
  facet_wrap(~target_type)
```

--------------------------------

### Example: plot_avail_forecasts with facet_wrap

Source: https://epiforecasts.io/scoringutils/reference/plot_avail_forecasts.html

Example that creates an availability plot from `avail_forecasts()` output, using `target_end_date` on the x-axis and faceting by `target_type`. The `show_numbers` argument is set to `FALSE` to hide counts.

```r
library(ggplot2)
avail_forecasts <- avail_forecasts(
  example_quantile, by = c("model", "target_type", "target_end_date")
)
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
plot_avail_forecasts(
 avail_forecasts, x = "target_end_date", show_numbers = FALSE
) +
 facet_wrap("target_type")
```

--------------------------------

### Example: Plotting a heatmap of bias

Source: https://epiforecasts.io/scoringutils/reference/plot_heatmap.html

This example demonstrates preparing quantile forecast scores and visualizing the bias metric across target types using plot_heatmap.

```R
scores <- example_quantile |>
  as_forecast_quantile() |>
  score()
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
scores <- summarise_scores(scores, by = c("model", "target_type"))
scores <- summarise_scores(
  scores, by = c("model", "target_type"),
  fun = signif, digits = 2
)

plot_heatmap(scores, x = "target_type", metric = "bias")
```

--------------------------------

### Validate forecast units with check_forecasts

Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html

Use set_forecast_unit to define the unique identifier columns for your forecasts and check_forecasts to validate the data structure. This example demonstrates an error triggered when forecasts are not uniquely identified.

```r
example_quantile %>%
  set_forecast_unit(
    c("location", "target_end_date",
      "forecast_date", "model", "horizon")
  ) %>%
  check_forecasts()
#> The following messages were produced when checking inputs:
#> 1.  144 values for `prediction` are NA in the data provided and the corresponding rows were removed. This may indicate a problem if unexpected.
#> Warning in check_forecasts(.): The following warnings were produced when checking inputs:
#> 1.  Some forecasts have different numbers of rows 
#> 2.  (e.g. quantiles or samples). 
#> 3.  scoringutils found: 
#> 4.  46, 23
#> 5.  . This may be a problem (it can potentially distort scores, 
#> 6.  making it more difficult to compare them), 
#> 7.  so make sure this is intended.
#> Error in check_forecasts(.): The following errors were produced when checking inputs:
#> 1.  There are instances with more than one forecast for the same target. This can't be right and needs to be resolved. Maybe you need to check the unit of a single forecast and add missing columns? Use the  function find_duplicates() to identify duplicate rows.
```

--------------------------------

### Identify duplicate forecasts with find_duplicates

Source: https://epiforecasts.io/scoringutils/articles/scoringutils.html

Use find_duplicates to investigate instances where multiple forecasts exist for the same target. This example filters the results to inspect specific median forecasts for the EuroCOVIDhub-ensemble.

```r
duplicates <- example_quantile %>%
  set_forecast_unit(
    c("location", "target_end_date",
      "forecast_date", "model", "horizon")
  ) %>%
  find_duplicates()

duplicates[quantile == 0.5 & model == "EuroCOVIDhub-ensemble", ] %>%
  head()
#>    true_value quantile prediction location target_end_date forecast_date
#> 1:     106987      0.5     119258       DE      2021-05-08    2021-05-03
#> 2:       1582      0.5       1568       DE      2021-05-08    2021-05-03
#> 3:      64985      0.5     110716       DE      2021-05-15    2021-05-03
#> 4:      64985      0.5      92649       DE      2021-05-15    2021-05-10
#> 5:       1311      0.5       1521       DE      2021-05-15    2021-05-03
#> 6:       1311      0.5       1422       DE      2021-05-15    2021-05-10
#>                    model horizon
#> 1: EuroCOVIDhub-ensemble       1
#> 2: EuroCOVIDhub-ensemble       1
#> 3: EuroCOVIDhub-ensemble       2
#> 4: EuroCOVIDhub-ensemble       1
#> 5: EuroCOVIDhub-ensemble       2
#> 6: EuroCOVIDhub-ensemble       1
```

--------------------------------

### Examples of get_metrics with binary forecasts

Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_binary.html

Demonstrates retrieving all default metrics, selecting specific metrics, and excluding specific metrics from the result.

```R
get_metrics(example_binary)
#> $brier_score
#> function (observed, predicted) 
#> {
#>     assert_input_binary(observed, predicted)
#>     observed <- as.numeric(observed) - 1
#>     brierscore <- (observed - predicted)^2
#>     return(brierscore)
#> }
#> <bytecode: 0x559a4d8df530>
#> <environment: namespace:scoringutils>
#> 
#> $log_score
#> function (observed, predicted) 
#> {
#>     assert_input_binary(observed, predicted)
#>     observed <- as.numeric(observed) - 1
#>     logs <- -log(1 - abs(observed - predicted))
#>     return(logs)
#> }
#> <bytecode: 0x559a4d8e2528>
#> <environment: namespace:scoringutils>
#> 
get_metrics(example_binary, select = "brier_score")
#> $brier_score
#> function (observed, predicted) 
#> {
#>     assert_input_binary(observed, predicted)
#>     observed <- as.numeric(observed) - 1
#>     brierscore <- (observed - predicted)^2
#>     return(brierscore)
#> }
#> <bytecode: 0x559a4d8df530>
#> <environment: namespace:scoringutils>
#> 
get_metrics(example_binary, exclude = "log_score")
#> $brier_score
#> function (observed, predicted) 
#> {
#>     assert_input_binary(observed, predicted)
#>     observed <- as.numeric(observed) - 1
#>     brierscore <- (observed - predicted)^2
#>     return(brierscore)
#> }
#> <bytecode: 0x559a4d8df530>
#> <environment: namespace:scoringutils>
#>
```

--------------------------------

### Install scoringutils from CRAN

Source: https://epiforecasts.io/scoringutils/index.html

Installs the stable CRAN version of the scoringutils package.

```R
install.packages("scoringutils")
```

--------------------------------

### Example: sample_to_quantile(example_integer)

Source: https://epiforecasts.io/scoringutils/reference/sample_to_quantile.html

Demonstrates converting the `example_integer` dataset to quantile format. The output shows a data.frame in long interval range format with columns including location, model, horizon, true_value, quantile, and prediction.

```r
sample_to_quantile(example_integer)
#>       location location_name target_end_date target_type forecast_date
#>    1:       DE       Germany      2021-01-02       Cases          <NA>
#>    2:       DE       Germany      2021-01-02       Cases          <NA>
#>    3:       DE       Germany      2021-01-02       Cases          <NA>
#>    4:       DE       Germany      2021-01-02       Cases          <NA>
#>    5:       DE       Germany      2021-01-02       Cases          <NA>
#>   ---                                                                 
#> 5151:       IT         Italy      2021-07-24      Deaths    2021-07-12
#> 5152:       IT         Italy      2021-07-24      Deaths    2021-07-12
#> 5153:       IT         Italy      2021-07-24      Deaths    2021-07-12
#> 5154:       IT         Italy      2021-07-24      Deaths    2021-07-12
#> 5155:       IT         Italy      2021-07-24      Deaths    2021-07-12
#>                      model horizon true_value quantile prediction
#>    1:                 <NA>      NA     127300     0.05         NA
#>    2:                 <NA>      NA     127300     0.25         NA
#>    3:                 <NA>      NA     127300     0.50         NA
#>    4:                 <NA>      NA     127300     0.75         NA
#>    5:                 <NA>      NA     127300     0.95         NA
#>   ---                                                            
#> 5151: epiforecasts-EpiNow2       2         78     0.05      44.90
#> 5152: epiforecasts-EpiNow2       2         78     0.25      89.25
#> 5153: epiforecasts-EpiNow2       2         78     0.50     151.50
#> 5154: epiforecasts-EpiNow2       2         78     0.75     208.00
#> 5155: epiforecasts-EpiNow2       2         78     0.95     469.10
```

--------------------------------

### Example: scoring quantile predictions with interval_score()

Source: https://epiforecasts.io/scoringutils/reference/interval_score.html

Example demonstrating how to compute interval scores for a vector of observed values and prediction intervals. The example generates random data and calls interval_score() with default settings.

```R
observed <- rnorm(30, mean = 1:30)
interval_range <- rep(90, 30)
alpha <- (100 - interval_range) / 100
lower <- qnorm(alpha / 2, rnorm(30, mean = 1:30))
upper <- qnorm((1 - alpha / 2), rnorm(30, mean = 11:40))

scoringutils:::interval_score(
  observed = observed,
  lower = lower,
  upper = upper,
  interval_range = interval_range
)
#>  [1] 0.7984288 0.6188294 0.7680146 0.7392137 2.4496739 0.6725337 0.5966822
#>  [8] 0.6659581 0.6581834 1.2866256 0.7423596 0.6155816 0.6176548 0.6484069
#> [15] 0.6584252 0.6929587 0.7558694 0.6724925 0.5716195 0.6251716 0.7431658
#> [22] 0.6906088 0.6524924 0.6274762 1.8011311 0.5840689 0.6602319 0.7076748
#> [29] 0.6060337 0.5491802
```

--------------------------------

### Example: compute correlations with get_correlations()

Source: https://epiforecasts.io/scoringutils/reference/get_correlations.html

Example workflow: score example_quantile data and then call get_correlations() to obtain a correlation matrix. The output shows correlations between metrics like wis, overprediction, underprediction, dispersion, bias, interval_coverage_50, interval_coverage_90, and ae_median.

```r
scores <- example_quantile |>
 as_forecast_quantile() |>
 score()
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.

get_correlations(scores)
#>           wis overprediction underprediction  dispersion        bias
#>         <num>          <num>           <num>       <num>       <num>
#> 1:  1.0000000     0.94297565      0.28377361  0.45566303  0.10545891
#> 2:  0.9429757     1.00000000     -0.03310356  0.32493799  0.21532161
#> 3:  0.2837736    -0.03310356      1.00000000  0.14580143 -0.35123801
#> 4:  0.4556630     0.32493799      0.14580143  1.00000000  0.11118365
#> 5:  0.1054589     0.21532161     -0.35123801  0.11118365  1.00000000
#> 6: -0.2076649    -0.14556039     -0.21392764 -0.09400664  0.01338140
#> 7: -0.4075613    -0.31824017     -0.35756699 -0.08614678  0.09802725
#> 8:  0.9886108     0.90326672      0.33589892  0.53809741  0.09578751
#>    interval_coverage_50 interval_coverage_90   ae_median               metric
#>                   <num>                <num>       <num>               <char>
#> 1:          -0.20766492          -0.40756133  0.98861080                  wis
#> 2:          -0.14556039          -0.31824017  0.90326672       overprediction
#> 3:          -0.21392764          -0.35756699  0.33589892      underprediction
#> 4:          -0.09400664          -0.08614678  0.53809741           dispersion
#> 5:           0.01338140           0.09802725  0.09578751                 bias
#> 6:           1.00000000           0.37245118 -0.24559356 interval_coverage_50
#> 7:           0.37245118           1.00000000 -0.41079097 interval_coverage_90
#> 8:          -0.24559356          -0.41079097  1.00000000            ae_median
```

--------------------------------

### Install development version from GitHub

Source: https://epiforecasts.io/scoringutils/index.html

Installs the unstable development version of scoringutils from GitHub with dependencies.

```R
remotes::install_github("epiforecasts/scoringutils", dependencies = TRUE)
```

--------------------------------

### set_forecast_unit() example with example_quantile

Source: https://epiforecasts.io/scoringutils/reference/set_forecast_unit.html

Example call to `set_forecast_unit()` using the `example_quantile` dataset, specifying the forecast unit columns. The output shows the resulting data.table with the forecast unit and key columns.

```r
example_quantile |>
  scoringutils:::set_forecast_unit(
    c("location", "target_end_date", "target_type", "horizon", "model")
  )
```

```output
#> Forecast type: quantile
#> Forecast unit:
#> location, target_end_date, target_type, horizon, and model
#> 
#> Key: <location, target_end_date, target_type>
#>        observed quantile_level predicted location target_end_date target_type
#>           <num>          <num>     <int>   <char>          <Date>      <char>
#>     1:   127300             NA        NA       DE      2021-01-02       Cases
#>     2:     4534             NA        NA       DE      2021-01-02      Deaths
#>     3:   154922             NA        NA       DE      2021-01-09       Cases
#>     4:     6117             NA        NA       DE      2021-01-09      Deaths
#>     5:   110183             NA        NA       DE      2021-01-16       Cases
#>    ---                                                                       
#> 20541:       78          0.850       352       IT      2021-07-24      Deaths
#> 20542:       78          0.900       397       IT      2021-07-24      Deaths
#> 20543:       78          0.950       499       IT      2021-07-24      Deaths
#> 20544:       78          0.975       611       IT      2021-07-24      Deaths
#> 20545:       78          0.990       719       IT      2021-07-24      Deaths
#>        horizon                model
#>          <num>               <char>
#>     1:      NA                 <NA>
#>     2:      NA                 <NA>
#>     3:      NA                 <NA>
#>     4:      NA                 <NA>
#>     5:      NA                 <NA>
#>    ---                             
#> 20541:       2 epiforecasts-EpiNow2
#> 20542:       2 epiforecasts-EpiNow2
#> 20543:       2 epiforecasts-EpiNow2
#> 20544:       2 epiforecasts-EpiNow2
#> 20545:       2 epiforecasts-EpiNow2
```

--------------------------------

### Example of get_metrics with ordinal forecast

Source: https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_ordinal.html

Retrieving the default scoring rules for an example ordinal forecast object.

```R
get_metrics(example_ordinal)
#> $log_score
#> function (observed, predicted, predicted_label) 
#> {
#>     assert_input_categorical(observed, predicted, predicted_label)
#>     n <- length(observed)
#>     if (n == 1) {
#>         predicted <- matrix(predicted, nrow = 1)
#>     }
#>     observed_indices <- as.numeric(observed)
#>     pred_for_observed <- predicted[cbind(1:n, observed_indices)]
#>     logs <- -log(pred_for_observed)
#>     return(logs)
#> }
#> <bytecode: 0x559a4f3839e8>
#> <environment: namespace:scoringutils>
#> 
#> $rps
#> function (observed, predicted, predicted_label) 
#> {
#>     assert_input_ordinal(observed, predicted, predicted_label)
#>     n <- length(observed)
#>     if (n == 1) {
#>         predicted <- matrix(predicted, nrow = 1)
#>     }
#>     correct_order <- as.numeric(predicted_label)
#>     ordered_predicted <- predicted[, correct_order]
#>     rps <- rps_probs(as.numeric(observed), ordered_predicted)
#>     return(rps)
#> }
#> <bytecode: 0x559a469eead0>
#> <environment: namespace:scoringutils>
#>
```

--------------------------------

### Compute coverage by model

Source: https://epiforecasts.io/scoringutils/reference/get_coverage.html

This example demonstrates how to convert a quantile forecast object and compute coverage metrics grouped by model.

```R
example_quantile |>
  as_forecast_quantile() |>
  get_coverage(by = "model")
#> ℹ Some rows containing NA values may be removed. This is fine if not
#>   unexpected.
#>                     model quantile_level interval_range interval_coverage
#>                    <char>          <num>          <num>             <num>
#>  1: EuroCOVIDhub-baseline          0.500              0       0.000000000
#>  2: EuroCOVIDhub-baseline          0.450             10       0.085937500
#>  3: EuroCOVIDhub-baseline          0.550             10       0.085937500
#>  4: EuroCOVIDhub-baseline          0.400             20       0.191406250
#>  5: EuroCOVIDhub-baseline          0.600             20       0.191406250
#>  6: EuroCOVIDhub-baseline          0.350             30       0.289062500
#>  7: EuroCOVIDhub-baseline          0.650             30       0.289062500
#>  8: EuroCOVIDhub-baseline          0.300             40       0.375000000
#>  9: EuroCOVIDhub-baseline          0.700             40       0.375000000
#> 10: EuroCOVIDhub-baseline          0.250             50       0.496093750
#> 11: EuroCOVIDhub-baseline          0.750             50       0.496093750
#> 12: EuroCOVIDhub-baseline          0.200             60       0.628906250
#> 13: EuroCOVIDhub-baseline          0.800             60       0.628906250
#> 14: EuroCOVIDhub-baseline          0.150             70       0.773437500
#> 15: EuroCOVIDhub-baseline          0.850             70       0.773437500
#> 16: EuroCOVIDhub-baseline          0.100             80       0.843750000
#> 17: EuroCOVIDhub-baseline          0.900             80       0.843750000
#> 18: EuroCOVIDhub-baseline          0.050             90       0.910156250
#> 19: EuroCOVIDhub-baseline          0.950             90       0.910156250
#> 20: EuroCOVIDhub-baseline          0.025             95       0.925781250
#> 21: EuroCOVIDhub-baseline          0.975             95       0.925781250
#> 22: EuroCOVIDhub-baseline          0.010             98       0.933593750
#> 23: EuroCOVIDhub-baseline          0.990             98       0.933593750
#> 24: EuroCOVIDhub-ensemble          0.500              0       0.003906250
#> 25: EuroCOVIDhub-ensemble          0.450             10       0.148437500
#> 26: EuroCOVIDhub-ensemble          0.550             10       0.148437500
#> 27: EuroCOVIDhub-ensemble          0.400             20       0.250000000
#> 28: EuroCOVIDhub-ensemble          0.600             20       0.250000000
#> 29: EuroCOVIDhub-ensemble          0.350             30       0.386718750
#> 30: EuroCOVIDhub-ensemble          0.650             30       0.386718750
#> 31: EuroCOVIDhub-ensemble          0.300             40       0.519531250
#> 32: EuroCOVIDhub-ensemble          0.700             40       0.519531250
#> 33: EuroCOVIDhub-ensemble          0.250             50       0.632812500
#> 34: EuroCOVIDhub-ensemble          0.750             50       0.632812500
#> 35: EuroCOVIDhub-ensemble          0.200             60       0.667968750
#> 36: EuroCOVIDhub-ensemble          0.800             60       0.667968750
#> 37: EuroCOVIDhub-ensemble          0.150             70       0.753906250
#> 38: EuroCOVIDhub-ensemble          0.850             70       0.753906250
#> 39: EuroCOVIDhub-ensemble          0.100             80       0.816406250
#> 40: EuroCOVIDhub-ensemble          0.900             80       0.816406250
#> 41: EuroCOVIDhub-ensemble          0.050             90       0.902343750
#> 42: EuroCOVIDhub-ensemble          0.950             90       0.902343750
#> 43: EuroCOVIDhub-ensemble          0.025             95       0.941406250
#> 44: EuroCOVIDhub-ensemble          0.975             95       0.941406250
#> 45: EuroCOVIDhub-ensemble          0.010             98       0.968750000
#> 46: EuroCOVIDhub-ensemble          0.990             98       0.968750000
#> 47:  epiforecasts-EpiNow2          0.500              0       0.004048583
#> 48:  epiforecasts-EpiNow2          0.450             10       0.093117409
#> 49:  epiforecasts-EpiNow2          0.550             10       0.093117409
#> 50:  epiforecasts-EpiNow2          0.400             20       0.165991903
#> 51:  epiforecasts-EpiNow2          0.600             20       0.165991903
#> 52:  epiforecasts-EpiNow2          0.350             30       0.230769231
#> 53:  epiforecasts-EpiNow2          0.650             30       0.230769231
```
Tokens:
Raw