### Install tidyposterior from GitHub

Source: https://tidyposterior.tidymodels.org/index.html

Installs the development version of the tidyposterior package from GitHub using the pak package manager.

```r
# install.packages("pak")
pak::pak("tidymodels/tidyposterior")
```

--------------------------------

### Minimal Runnable Code Example

Source: https://tidyposterior.tidymodels.org/ISSUE_TEMPLATE.html

A minimal, runnable R code example demonstrating data preparation and model fitting using tidyposterior. This code is intended for issue reporting.

```r
library(tidyposterior)
library(dplyr)

data(precise_example)

accuracy <- precise_example %>%
   select(id, contains("Accuracy")) %>%
   setNames(tolower(gsub("_Accuracy$", "", names(.)))) 
accuracy

acc_model <- perf_mod(accuracy, seed = 13311, verbose = FALSE)   
```

--------------------------------

### Install tidyposterior Package

Source: https://tidyposterior.tidymodels.org/ISSUE_TEMPLATE.html

Installs the latest version of the tidyposterior package. Ensure you are starting a new R session before installation.

```r
update.packages(oldPkgs="tidyposterior", ask=FALSE)
```

--------------------------------

### Install tidyposterior from CRAN

Source: https://tidyposterior.tidymodels.org/index.html

Installs the released version of the tidyposterior package from CRAN.

```r
install.packages("tidyposterior")
```

--------------------------------

### Setup Resampling Data

Source: https://tidyposterior.tidymodels.org/index.html

Initialize the tidymodels environment and create 10-fold cross-validation folds for a classification dataset.

```R
library(tidymodels)
library(tidyposterior)

data(two_class_dat, package = "modeldata")

set.seed(100)
folds <- vfold_cv(two_class_dat)
```

--------------------------------

### Prepare Resampling Data

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Setup for 10-fold cross-validation data using tidymodels.

```R
   library(tidymodels)
   library(tidyposterior)
   library(workflowsets)

   data(two_class_dat, package = "modeldata")

   set.seed(100)
   folds <- vfold_cv(two_class_dat)
```

--------------------------------

### Define Bayesian Model Formulas

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Examples of formulas used to specify the Bayesian model structure for performance metric comparison.

```R
   # One ID field and common variance:
     statistic ~ model + (model | id)

   # One ID field and heterogeneous variance:
     statistic ~ model + (model + 0 | id)

   # Repeated CV (id = repeat, id2 = fold within repeat)
   # with a common variance:
     statistic ~ model + (model | id2/id)

   # Repeated CV (id = repeat, id2 = fold within repeat)
   # with a heterogeneous variance:
     statistic ~ model + (model + 0| id2/id)

   # Default for unknown resampling method and
   # multiple ID fields:
     statistic ~ model + (model | idN/../id)
```

--------------------------------

### Define Logistic Regression Model Specification

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Specifies a logistic regression model using the 'glm' engine. This is a basic model setup.

```R
logistic_spec <- logistic_reg() %>% set_engine("glm")
```

--------------------------------

### Summarize ROC Statistics

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Calculates overall ROC statistics by averaging splines and spatial sign values. This is useful for getting a single performance metric per method.

```R
summarize(
  roc_values,
  splines = mean(splines),
  spatial_sign = mean(spatial_sign)
)
```

--------------------------------

### Create a Workflow Set

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Initialize a workflow set using `workflow_set()` to manage and evaluate multiple model configurations. This involves defining different model formulas and specifying the modeling engine.

```R
library(workflowsets)

logistic_set <-
  workflow_set(
    list(A = Class ~ A, B = Class ~ B, ratio = Class ~ I(log(A/B)), 
         spatial_sign = spatial_sign_rec),
    list(logistic = logistic_spec)
  )
logistic_set
```

--------------------------------

### Load and Display precise_example Data

Source: https://tidyposterior.tidymodels.org/reference/precise_example.html

Loads the 'precise_example' data set into the R environment and displays its contents. This data set contains results from a 10-fold cross-validation of a classification analysis.

```r
data(precise_example)
precise_example
```

--------------------------------

### Create Workflow Sets

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Organize multiple workflows into a collection using workflow_set for batch evaluation.

```R
library(workflowsets)

logistic_set <- 
  workflow_set(
    list(A = Class ~ A, B = Class ~ B, ratio = Class ~ I(log(A/B)), 
         spatial_sign = spatial_sign_rec),
    list(logistic = logistic_spec)
  )
logistic_set
```

--------------------------------

### Train Models with caret

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Create equivalent models using the caret package for comparison.

```R
   library(caret)

   set.seed(102)
   logistic_caret <- train(Class ~ ., data = two_class_dat, method = "glm",
                           trControl = trainControl(method = "cv"))

   set.seed(102)
   mars_caret <- train(Class ~ ., data = two_class_dat, method = "gcvEarth",
                       tuneGrid = data.frame(degree = 1),
                       trControl = trainControl(method = "cv"))
```

--------------------------------

### Create Reproducible Dataset with dput

Source: https://tidyposterior.tidymodels.org/ISSUE_TEMPLATE.html

Generates a minimal, reproducible dataset using `dput()` for R. This is crucial for debugging and issue reporting. Use `droplevels()` if dealing with factors with many levels.

```r
dput(head(iris,4))
```

```r
dput(droplevels(head(iris, 4)))
```

--------------------------------

### Use Workflow Set as Input

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Pass a collection of models and preprocessing steps using a workflow set object.

```R
   example_wset <-
     as_workflow_set(logistic = logistic_reg_glm_res, mars = mars_earth_res)

   set.seed(101)
   roc_model_via_wflowset <- perf_mod(example_wset, refresh = 0)
   tidy(roc_model_via_rset) %>% summary()
```

--------------------------------

### Contrast Model Performance

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Compute differences between specific model configurations using contrast_models and summarize or visualize the results.

```R
grid_diff <-
  contrast_models(
    grid_mod,
    list_1 = rep("Preprocessor1_Model1", 3),
    list_2 = c(
      "Preprocessor2_Model1", # <-  3 df spline
      "Preprocessor3_Model1", # <-  5 df spline
      "Preprocessor4_Model1"  # <- 10 df spline
    ),
    seed = 7
  )
```

```R
summary(grid_diff)
```

```R
autoplot(grid_diff)
```

```R
autoplot(grid_diff, size = 0.02)
```

--------------------------------

### Define Recipe for Spline Model Tuning

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Use `tune()` to mark parameters for optimization in a recipe. This prepares the model for tuning by specifying which parameters will be varied.

```R
spline_rec <- 
  recipe(Class ~ ., data = two_class_dat) %>% 
  step_ns(A, B, deg_free = tune())
```

--------------------------------

### contrast_models()

Source: https://tidyposterior.tidymodels.org/reference/index.html

Estimates the difference between models based on posterior distributions.

```APIDOC
## contrast_models()

### Description
Estimate the Difference Between Models.

### Method
Function Call

### Endpoint
contrast_models(object, ...)
```

--------------------------------

### Transformation Object Usage

Source: https://tidyposterior.tidymodels.org/reference/transformations.html

Demonstrates applying a transformation function and its inverse using the logit_trans object.

```R
logit_trans$func(.5)
#> [1] 0
logit_trans$inv(0)
#> [1] 0.5
```

--------------------------------

### Contrast Model Performance Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Computes posterior distributions of performance differences between models. The `summary()` function then provides statistics on these differences, including probability, mean, and credible intervals.

```R
preproc_diff <- contrast_models(rset_mod, seed = 4)
summary(preproc_diff, seed = 5)
```

--------------------------------

### Rank and Visualize Results

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Ranks workflow performance and generates plots for comparison.

```R
rank_results(logistic_res, rank_metric = "roc_auc") %>% 
  filter(.metric == "roc_auc")
```

```R
autoplot(logistic_res, metric = "roc_auc")
```

--------------------------------

### Create Spatial Sign Workflow

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Constructs a tidymodels workflow that includes a recipe for spatial sign transformation and normalization, combined with a logistic regression model specification.

```R
spatial_sign_wflow <- 
  workflow() %>% 
  add_recipe(spatial_sign_rec) %>% 
  add_model(logistic_spec)
```

--------------------------------

### Visualize Data After Spatial Sign Transformation

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Applies normalization and spatial sign transformation to predictors 'A' and 'B' using a recipe, then bakes the recipe and visualizes the transformed data. This helps understand the effect of the transformation.

```R
spatial_sign_rec <- 
  recipe(Class ~ ., data = two_class_dat) %>% 
  step_normalize(A, B) %>% 
  step_spatialsign(A, B)

spatial_sign_rec %>% 
  prep() %>% 
  bake(new_data = NULL) %>% 
  ggplot(aes(x = A, y = B, col = Class)) + 
  geom_point(alpha = 0.3, cex = 2) + 
  coord_fixed()
```

--------------------------------

### Visualize Workflow Set Performance

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Generate a plot to visualize the performance metrics across different workflows in a set using `autoplot`. Specify the metric of interest for the visualization.

```R
autoplot(logistic_res, metric = "roc_auc")
```

--------------------------------

### Perform Grid Search for Model Tuning

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Evaluate multiple parameter values using tune_grid() and collect the resulting metrics.

```R
spline_tune <-
  logistic_spec %>%
  tune_grid(
    spline_rec,
    resamples = cv_folds,
    grid = tibble(deg_free = c(1, 3, 5, 10)),
    metrics = metric_set(roc_auc),
    control = control_grid(save_workflow = TRUE)
  )
collect_metrics(spline_tune) %>% 
  arrange(desc(mean))
```

--------------------------------

### Visualize Posterior Distributions

Source: https://tidyposterior.tidymodels.org/reference/autoplot.posterior_diff.html

Use this function to create a density plot for each contrast in a faceted grid. Requires the `ggplot2` library.

```R
data(ex_objects)
library(ggplot2)
autoplot(contrast_samples)
```

--------------------------------

### Visualize Model Contrast Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Visualize the results of model contrast analysis using `autoplot()`. This can help in interpreting the practical significance of performance differences between models.

```R
autoplot(grid_diff)
```

--------------------------------

### Evaluate Workflow Sets

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Uses workflow_map to evaluate multiple workflows that do not require tuning.

```R
logistic_res <- 
  logistic_set %>% 
  workflow_map("fit_resamples", seed = 3, resamples = cv_folds, 
               metrics = metric_set(roc_auc)) 
logistic_res
```

--------------------------------

### Create Spline Recipe and Workflow

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Defines a recipe for logistic regression that includes a spline basis expansion for predictors 'A' and 'B' with 3 degrees of freedom. It then adds this recipe to a workflow with the logistic regression model specification.

```R
spline_rec <- 
  recipe(Class ~ ., data = two_class_dat) %>% 
  step_ns(A, B, deg_free = 3)

spline_wflow <- 
  workflow() %>% 
  add_recipe(spline_rec) %>% 
  add_model(logistic_spec)
```

--------------------------------

### Contrast Sub-model Performance

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Compare specific sub-models by computing their performance differences using `contrast_models()`. This is useful for understanding the impact of parameter choices, such as degrees of freedom in splines.

```R
grid_diff <-
  contrast_models(
    grid_mod,
    list_1 = rep("Preprocessor1_Model1", 3),
    list_2 = c(
      "Preprocessor2_Model1", # <-  3 df spline
      "Preprocessor3_Model1", # <-  5 df spline
      "Preprocessor4_Model1"  # <- 10 df spline
    ),
    seed = 7
  )
```

--------------------------------

### Visualizing ROPE Analysis for Model Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Creates a visual representation of the ROPE analysis for model differences, showing the distribution of posterior differences relative to the specified practical equivalence region.

```r
autoplot(preproc_diff, size = 0.02)
```

--------------------------------

### Contrast Models Function Signature

Source: https://tidyposterior.tidymodels.org/reference/contrast_models.html

This is the function signature for `contrast_models`. It takes an object from `perf_mod()` and optional lists to define pairwise contrasts.

```R
contrast_models(x, list_1 = NULL, list_2 = NULL, seed = sample.int(10000, 1))
```

--------------------------------

### Visualize ROPE Analysis for Model Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Visualize the Region Of Practical Equivalence (ROPE) for model contrast results. This plot helps determine if the observed differences are practically significant.

```R
autoplot(grid_diff, size = 0.02)
```

--------------------------------

### Summarize Model Contrasts

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Generate a summary of the contrasts computed between models. This provides detailed statistics on the differences in performance, including probabilities and mean differences.

```R
summary(grid_diff)
```

--------------------------------

### Append Results to Workflow Set

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Converts existing tuned results to a workflow set and merges them with current results.

```R
logistic_res <- 
  logistic_res %>% 
  bind_rows(
    as_workflow_set(splines = spline_tune)
  )
logistic_res
```

--------------------------------

### Execute perf_mod with caret resamples

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Use the caret resamples function to collect results before passing to perf_mod.

```R
   caret_resamples <- resamples(list(logistic = logistic_caret, mars = mars_caret))

   set.seed(101)
   roc_model_via_caret <- perf_mod(caret_resamples, refresh = 0)
   tidy(roc_model_via_caret) %>% summary()
```

--------------------------------

### Gather Session Information

Source: https://tidyposterior.tidymodels.org/ISSUE_TEMPLATE.html

Collects session information for R, which is essential for debugging and reporting issues. Use `sessionInfo()` or `sessioninfo::session_info()`.

```r
sessionInfo()

# or sessioninfo::session_info()
```

--------------------------------

### Append Tuned Spline Results to Workflow Set

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Combine results from a tuned model (e.g., splines) with an existing workflow set by converting the tuned object to a workflow set and using `bind_rows`. This allows for unified analysis.

```R
logistic_res <- 
  logistic_res %>%
  bind_rows(
    as_workflow_set(splines = spline_tune)
  )
logistic_res
```

--------------------------------

### Visualize Posterior Distributions with ggplot2

Source: https://tidyposterior.tidymodels.org/index.html

Use the tidy() method to extract posterior distributions and visualize them using ggplot2. This is useful for understanding the uncertainty in model parameters.

```R
roc_model_via_df %>/
  tidy() %>/
  ggplot(aes(x = posterior)) + 
  geom_histogram(bins = 40, col = "blue", fill = "blue", alpha = .4) + 
  facet_wrap(~ model, ncol = 1) + 
  xlab("Area Under the ROC Curve")
```

--------------------------------

### Collect Metrics

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Extracts and filters performance metrics from the workflow set results.

```R
collect_metrics(logistic_res) %>% 
  filter(.metric == "roc_auc")
```

--------------------------------

### Perform Bayesian Model Comparison

Source: https://tidyposterior.tidymodels.org/index.html

Pass the prepared metrics data frame to perf_mod to perform Bayesian analysis of model performance.

```R
set.seed(101)
roc_model_via_df <- perf_mod(resamples_df, iter = 2000)
```

--------------------------------

### perf_mod Function Documentation

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Documentation for the perf_mod function and its methods, used for Bayesian analysis of resampling statistics.

```APIDOC
## perf_mod

### Description
Bayesian analysis used here to answer the question: "when looking at resampling results, are the differences between models 'real?'" To answer this, a model can be created were the _outcome_ is the resampling statistics (e.g. accuracy or RMSE). These values are explained by the model types. In doing this, we can get parameter estimates for each model's affect on performance and make statistical (and practical) comparisons between models.

### Usage
```R
perf_mod(object, ...)

# S3 method for rset
perf_mod(object, transform = no_trans, hetero_var = FALSE, formula = NULL, ...)

# S3 method for resamples
perf_mod(object, transform = no_trans, hetero_var = FALSE, metric = object$metrics[1], ...)

# S3 method for data.frame
perf_mod(object, transform = no_trans, hetero_var = FALSE, formula = NULL, ...)

# S3 method for tune_results
perf_mod(object, metric = NULL, transform = no_trans, hetero_var = FALSE, formula = NULL, filter = NULL, ...)

# S3 method for workflow_set
perf_mod(object, metric = NULL, transform = no_trans, hetero_var = FALSE, formula = NULL, ...)
```

### Arguments 

* `object`
  Depending on the context (see Details below):
  * A data frame with `id` columns for the resampling groupds and metric results in all of the other columns..
  * An `rset` object (such as `rsample::vfold_cv()`) containing the `id` column(s) and at least two numeric columns of model performance statistics (e.g. accuracy).
  * An object from `caret::resamples`.
  * An object with class `tune_results`, which could be produced by `tune::tune_grid()`, `tune::tune_bayes()` or similar.
  * A workflow set where all results contain the metric value given in the `metric` argument value.

* `...`
  Additional arguments to pass to `rstanarm::stan_glmer()` such as `verbose`, `prior`, `seed`, `refresh`, `family`, etc.

* `transform`
  An named list of transformation and inverse transformation functions. See `logit_trans()` as an example.

* `hetero_var`
  A logical; if `TRUE`, then different variances are estimated for each model group. Otherwise, the same variance is used for each group. Estimating heterogeneous variances may slow or prevent convergence.

* `formula`
  An optional model formula to use for the Bayesian hierarchical model (see Details below).

* `metric`
  A single character value for the statistic from the `resamples` object that should be analyzed.

* `filter`
  A conditional logic statement that can be used to filter the statistics generated by `tune_results` using the tuning parameter values or the `.config` column.

### Value
An object of class `perf_mod`. If a workfkow set is given in `object`, there is an extra class of `"perf_mod_workflow_set"`.

### Details
These functions can be used to process and analyze matched resampling statistics from different models using a Bayesian generalized linear model with effects for the model and the resamples.
```

--------------------------------

### Prepare Data Frame for perf_mod

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Extract metrics from model results and join them into a single data frame for comparison.

```R
   logistic_roc <-
     collect_metrics(logistic_reg_glm_res, summarize = FALSE) %>%
     dplyr::filter(.metric == "roc_auc") %>%
     dplyr::select(id, logistic = .estimate)

   mars_roc <-
     collect_metrics(mars_earth_res, summarize = FALSE) %>%
     dplyr::filter(.metric == "roc_auc") %>%
     dplyr::select(id, mars = .estimate)

   resamples_df <- full_join(logistic_roc, mars_roc, by = "id")
   resamples_df
```

--------------------------------

### Summarize Bayesian ANOVA Model Output

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Prints a detailed summary of the fitted Bayesian ANOVA model, including model information, estimates for intercept and model terms, and group-specific random effects.

```R
print(summary(rset_mod), digits = 3)
```

--------------------------------

### tidy()

Source: https://tidyposterior.tidymodels.org/reference/index.html

Extracts posterior distributions for models.

```APIDOC
## tidy()

### Description
Extract Posterior Distributions for Models.

### Method
Function Call

### Endpoint
tidy(x, ...)
```

--------------------------------

### Visualize Bayesian Results

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Generates plots for Bayesian model comparisons, including credible intervals and ROPE estimates.

```R
autoplot(roc_mod)
```

```R
autoplot(roc_mod, type = "ROPE", size = 0.025)
```

--------------------------------

### Perform Bayesian Analysis

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Uses perf_mod to compare the best submodels from each workflow using Bayesian methods.

```R
roc_mod <- perf_mod(logistic_res, metric = "roc_auc", seed = 1, refresh = 0)
```

--------------------------------

### Execute perf_mod with Data Frame

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Pass the prepared data frame to perf_mod to perform Bayesian model comparison.

```R
   set.seed(101)
   roc_model_via_df <- perf_mod(resamples_df, refresh = 0)
   tidy(roc_model_via_df) %>% summary()
```

--------------------------------

### Summarize Posterior Distributions

Source: https://tidyposterior.tidymodels.org/reference/summary.posterior.html

This function creates numerical summaries for each model, including the posterior mean and upper and lower credible intervals (aka uncertainty intervals).

```APIDOC
## summary.posterior

### Description
Numerical summaries are created for each model including the posterior mean and upper and lower credible intervals (aka uncertainty intervals).

### Usage
```R
summary(object, prob = 0.9, seed = sample.int(10000, 1), ...)
```

### Arguments 

* **object** (object) - An object produced by `tidy.perf_mod()`.
* **prob** (number) - A number p (0 < p < 1) indicating the desired probability mass to include in the intervals.
* **seed** (integer) - A single integer for sampling from the posterior.
* **...** - Not currently used

### Value
A data frame with summary statistics and a row for each model.

### Examples

```R
data("ex_objects")

summary(posterior_samples)
```
```

--------------------------------

### Prepare Metrics for Comparison

Source: https://tidyposterior.tidymodels.org/index.html

Extract ROC AUC metrics from resampled results and join them into a single data frame for comparison.

```R
logistic_roc <- 
  collect_metrics(logistic_reg_glm_res, summarize = FALSE) %>% 
  dplyr::filter(.metric == "roc_auc") %>% 
  dplyr::select(id, logistic = .estimate)

mars_roc <- 
  collect_metrics(mars_earth_res, summarize = FALSE) %>% 
  dplyr::filter(.metric == "roc_auc") %>% 
  dplyr::select(id, mars = .estimate)

resamples_df <- full_join(logistic_roc, mars_roc, by = "id")
resamples_df
```

--------------------------------

### Visualize Posterior Distributions

Source: https://tidyposterior.tidymodels.org/reference/autoplot.posterior.html

Generates a plot of posterior distributions for model statistics. This function is applicable to objects of classes 'posterior' and 'perf_mod'.

```R
data(ex_objects)
autoplot(posterior_samples)
```

--------------------------------

### autoplot.posterior and autoplot.perf_mod

Source: https://tidyposterior.tidymodels.org/reference/autoplot.posterior.html

Generates a simple plot of posterior distributions for objects of classes 'posterior' and 'perf_mod'.

```APIDOC
## autoplot.posterior and autoplot.perf_mod

### Description
For objects of classes `posterior` and `perf_mod`, `autoplot()` produces a simple plot of posterior distributions.

### Method
S3 method

### Endpoint
N/A (R function)

### Parameters
#### Arguments
- **object** (object) - An object produced by `perf_mod()`, `tidy.perf_mod()`, or a workflow set with computed results.
- **...** (...) - Options passed to `geom_line(stat = "density", ...)`.

### Request Example
```R
data(ex_objects)
autoplot(posterior_samples)
```

### Response
#### Success Response
- **ggplot2::ggplot()** - A ggplot2::ggplot() object.

#### Response Example
(A ggplot2 object is returned, visualization depends on the input data)
```

--------------------------------

### Analyze Model Performance from Tuning

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Use `perf_mod()` to analyze the performance of models resulting from a tuning process. This function can be used with objects from tuning functions and allows for detailed performance computations.

```R
grid_mod <- perf_mod(spline_tune, seed = 6, iter = 5000, chains = 5, refresh = 0)
autoplot(grid_mod)
```

--------------------------------

### Use rsample Object as Input

Source: https://tidyposterior.tidymodels.org/reference/perf_mod.html

Merge results back into an rsample object to preserve model formula information for perf_mod.

```R
   resamples_rset <-
     full_join(folds, logistic_roc, by = "id") %>%
     full_join(mars_roc, by = "id")

   set.seed(101)
   roc_model_via_rset <- perf_mod(resamples_rset, refresh = 0)
   tidy(roc_model_via_rset) %>% summary()
```

--------------------------------

### contrast_models Function

Source: https://tidyposterior.tidymodels.org/reference/contrast_models.html

Computes the posterior distribution of the difference between models.

```APIDOC
## contrast_models

### Description
The posterior distributions created by perf_mod() can be used to obtain the posterior distribution of the difference(s) between models. One or more comparisons can be computed at the same time.

### Usage
contrast_models(x, list_1 = NULL, list_2 = NULL, seed = sample.int(10000, 1))

### Parameters
- **x** (object) - Required - An object produced by perf_mod().
- **list_1** (character vector) - Optional - Specifies the first part of the pairwise contrast. Parameterized as list_1[i] - list_2[i].
- **list_2** (character vector) - Optional - Specifies the second part of the pairwise contrast. Parameterized as list_1[i] - list_2[i].
- **seed** (integer) - Optional - A single integer for sampling from the posterior.

### Value
A data frame of the posterior distribution(s) of the difference(s). The object has an extra class of "posterior_diff".
```

--------------------------------

### Define Model Specifications

Source: https://tidyposterior.tidymodels.org/index.html

Create model specifications for logistic regression and MARS models.

```R
logistic_reg_glm_spec <-
  logistic_reg() %>%
  set_engine('glm')

mars_earth_spec <-
  mars(prod_degree = 1) %>%
  set_engine('earth') %>%
  set_mode('classification')
```

--------------------------------

### Summarize Posterior Distributions

Source: https://tidyposterior.tidymodels.org/reference/summary.posterior.html

Use this function to generate a data frame with summary statistics (mean, credible intervals) for posterior distributions of model parameters. Ensure the input object is produced by `tidy.perf_mod()`.

```r
summary(object, prob = 0.9, seed = sample.int(10000, 1), ...)
```

```r
data("ex_objects")

summary(posterior_samples)
#> # A tibble: 3 × 4
#>   model         mean lower upper
#>   <chr>        <dbl> <dbl> <dbl>
#> 1 cart         0.851 0.826 0.875
#> 2 logistic_reg 0.883 0.859 0.909
#> 3 mars         0.884 0.859 0.910
```

--------------------------------

### Posterior Class Compatibility Methods

Source: https://tidyposterior.tidymodels.org/reference/vctrs_methods_posterior.html

These methods are used to ensure that objects of class 'posterior' are compatible with dplyr verbs. They handle restoration, proxy creation, type-2 prototyping, and casting between 'posterior', 'tbl_df', and 'data.frame' classes.

```R
vec_restore.posterior(x, to, ...)
```

```R
vec_proxy.posterior(x, ...)
```

```R
vec_ptype2.posterior.posterior(x, y, ..., x_arg = "", y_arg = "")
```

```R
vec_ptype2.posterior.tbl_df(x, y, ..., x_arg = "", y_arg = "")
```

```R
vec_ptype2.tbl_df.posterior(x, y, ..., x_arg = "", y_arg = "")
```

```R
vec_ptype2.posterior.data.frame(x, y, ..., x_arg = "", y_arg = "")
```

```R
vec_ptype2.data.frame.posterior(x, y, ..., x_arg = "", y_arg = "")
```

```R
vec_cast.posterior.posterior(x, to, ..., x_arg = "", to_arg = "")
```

```R
vec_cast.posterior.tbl_df(x, to, ..., x_arg = "", to_arg = "")
```

```R
vec_cast.tbl_df.posterior(x, to, ..., x_arg = "", to_arg = "")
```

```R
vec_cast.posterior.data.frame(x, to, ..., x_arg = "", to_arg = "")
```

```R
vec_cast.data.frame.posterior(x, to, ..., x_arg = "", to_arg = "")
```

--------------------------------

### Summarizing ROPE Analysis for Model Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Computes and displays the probability of posterior differences falling within, below, and above the specified region of practical equivalence (ROPE). This helps quantify practical significance.

```r
summary(preproc_diff, size = 0.02) %>%
  select(contrast, starts_with("pract"))
```

--------------------------------

### Summarize Posterior Distributions of Model Differences

Source: https://tidyposterior.tidymodels.org/reference/summary.posterior_diff.html

This function computes credible intervals for the differences between model posterior distributions and calculates Region of Practical Equivalence (ROPE) statistics when an effective size of a difference is provided.

```APIDOC
## POST /websites/tidyposterior_tidymodels

### Description
Summarizes posterior distributions of model differences by computing credible intervals and ROPE statistics.

### Method
POST

### Endpoint
/websites/tidyposterior_tidymodels

### Parameters
#### Query Parameters
- **prob** (number) - Optional - A number p (0 < p < 1) indicating the desired probability mass to include in the intervals.
- **size** (number) - Optional - The size of an effective difference in the units of the chosen metric. For example, a 5 percent increase in accuracy (`size = 0.05`) between two models might be considered a "real" difference.

### Request Body
- **object** (object) - Required - An object produced by `contrast_models()`.

### Request Example
```json
{
  "object": {},
  "prob": 0.9,
  "size": 0
}
```

### Response
#### Success Response (200)
- **data.frame** (data frame) - A data frame with interval and ROPE statistics for each comparison.

#### Response Example
```json
{
  "data.frame": [
    {
      "contrast": "logistic_regression",
      "probability": 1,
      "mean": 0.0329,
      "lower": 0.0196,
      "upper": 0.0462,
      "size": 0,
      "pract_neg": null,
      "pract_equiv": null,
      "pract_pos": null
    },
    {
      "contrast": "logistic_regression",
      "probability": 0.472,
      "mean": -0.000540,
      "lower": -0.0142,
      "upper": 0.0130,
      "size": 0,
      "pract_neg": null,
      "pract_equiv": null,
      "pract_pos": null
    }
  ]
}
```
```

--------------------------------

### Transformation Functions Overview

Source: https://tidyposterior.tidymodels.org/reference/transformations.html

A set of objects designed to facilitate the use of outcome transformations for modeling. These are useful when dealing with statistics that have natural bounds or skewed distributions.

```APIDOC
## Transformation Functions

### Description
Provides functions to transform outcome variables prior to modeling, especially useful for statistics with natural bounds (e.g., 0 to 1 for accuracy) or skewed distributions. This helps in assuming appropriate priors and ensuring posterior estimates remain within valid ranges.

### Available Transformations

- `no_trans`: No transformation applied.
- `logit_trans`: Useful for statistics bounded between 0 and 1 (e.g., accuracy, ROC AUC).
- `ln_trans`: Useful for statistics that are right-skewed and strictly positive.
- `inv_trans`: Useful for statistics that are right-skewed and strictly positive.
- `Fisher_trans`: Originally for correlation statistics, but applicable to metrics between -1 and 1 (e.g., Kappa).

### Usage Examples

#### `logit_trans`
```R
logit_trans$func(.5)
#> [1] 0
logit_trans$inv(0)
#> [1] 0.5
```

### Details
- `logit_trans` is suitable for metrics like accuracy or AUC, which are naturally bounded between 0 and 1.
- `ln_trans` and `inv_trans` are beneficial for positively skewed data.
- `Fisher_trans` can be used for metrics like Kappa, which range from -1 to 1.
```

--------------------------------

### Fit Resamples for Logistic Regression Workflows

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Use `workflow_map` to efficiently fit resamples for multiple logistic regression models within a workflow set. This function is suitable when no tuning is required. Ensure `cv_folds` and `metric_set` are defined.

```R
logistic_res <- 
  logistic_set %>%
  workflow_map("fit_resamples", seed = 3, resamples = cv_folds, 
               metrics = metric_set(roc_auc)) 
logistic_res
```

--------------------------------

### Extract Posterior Samples

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Retrieves posterior samples of performance metrics for each model. Requires a seed for reproducibility of the sampling process.

```R
tidy(rset_mod, seed = 3)
```

--------------------------------

### Visualize Two-Class Data

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Plots the 'A' and 'B' predictors against each other, colored by 'Class', to visualize the separation of two classes. Requires 'tidymodels' and 'tidyposterior' libraries.

```R
library(tidymodels)
library(tidyposterior)

data(two_class_dat)

ggplot(two_class_dat, aes(x = A, y = B, col = Class)) + 
  geom_point(alpha = 0.3, cex = 2) + 
  coord_fixed()
```

--------------------------------

### Tidy Posterior for perf_mod Objects

Source: https://tidyposterior.tidymodels.org/reference/tidy.perf_mod.html

Use this S3 method for `perf_mod` objects to create a data frame of posterior predictive distribution values. It samples from the posterior based on the provided seed. Note that this posterior only reflects group variability (fixed effects).

```r
tidy(x, seed = sample.int(10000, 1), ...)
```

--------------------------------

### Create Cross-Validation Folds

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Generates 10-fold cross-validation folds for a given dataset. Ensure 'rsample' package is available. Sets a seed for reproducibility.

```R
set.seed(1)
cv_folds <- vfold_cv(two_class_dat)
cv_folds
```

--------------------------------

### Summarize Posterior Differences

Source: https://tidyposterior.tidymodels.org/reference/summary.posterior_diff.html

Use this function to generate credible intervals and ROPE statistics for model comparisons. Specify the desired probability mass for intervals and the size of a practically meaningful difference.

```R
summary(object, prob = 0.9, size = 0, ...)
```

```R
data("ex_objects")

summary(contrast_samples)
```

```R
summary(contrast_samples, size = 0.025)
```

--------------------------------

### Tune Spline Model using Grid Search

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Evaluate multiple values of a tuning parameter (e.g., `deg_free`) using `tune_grid()`. This function resamples the data for each parameter value and computes specified metrics, such as ROC AUC.

```R
spline_tune <-
  logistic_spec %>%
  tune_grid(
    spline_rec,
    resamples = cv_folds,
    grid = tibble(deg_free = c(1, 3, 5, 10)),
    metrics = metric_set(roc_auc),
    control = control_grid(save_workflow = TRUE)
  )
collect_metrics(spline_tune) %>%
  arrange(desc(mean))
```

--------------------------------

### autoplot.posterior_diff

Source: https://tidyposterior.tidymodels.org/reference/autoplot.posterior_diff.html

Generates a density plot for each contrast in a faceted grid, visualizing the posterior distributions of model differences.

```APIDOC
## POST /autoplot.posterior_diff

### Description
Visualizes the posterior distributions of model differences by creating a density plot for each contrast, faceted by the models being contrasted.

### Method
POST

### Endpoint
/autoplot.posterior_diff

### Parameters
#### Arguments
- **object** (posterior_diff) - Required - An object produced by `contrast_models()`.
- **size** (numeric) - Optional - The size of an effective difference. For example, a 5"real" difference.
- **...** (any) - Optional - Options passed to `geom_line(stat = "density", ...)`.

### Request Example
```json
{
  "object": "<posterior_diff object>",
  "size": 0
}
```

### Response
#### Success Response (200)
- **ggplot2::ggplot()** (ggplot) - A ggplot2 object using geom_density, faceted by the models being contrasted.

#### Response Example
```json
{
  "plot": "<ggplot object>"
}
```
```

--------------------------------

### vctrs Methods for posterior_diff

Source: https://tidyposterior.tidymodels.org/reference/vctrs_methods_posterior_diff.html

Methods for managing the posterior_diff class, which is a tibble requiring specific columns: difference (numeric), model_1 (character), model_2 (character), and contrast (character).

```APIDOC
## vctrs Methods for posterior_diff

### Description
These methods implement vctrs protocols to ensure that objects of class `posterior_diff` maintain their required column structure during data manipulation. If operations violate these requirements, the object is down-cast to a standard tibble.

### Methods
- vec_restore.posterior_diff(x, to, ...)
- vec_proxy.posterior_diff(x, ...)
- vec_ptype2.posterior_diff.posterior_diff(x, y, ..., x_arg, y_arg)
- vec_ptype2.posterior_diff.tbl_df(x, y, ..., x_arg, y_arg)
- vec_ptype2.tbl_df.posterior_diff(x, y, ..., x_arg, y_arg)
- vec_ptype2.posterior_diff.data.frame(x, y, ..., x_arg, y_arg)
- vec_ptype2.data.frame.posterior_diff(x, y, ..., x_arg, y_arg)
- vec_cast.posterior_diff.posterior_diff(x, to, ..., x_arg, to_arg)
- vec_cast.posterior_diff.tbl_df(x, to, ..., x_arg, to_arg)
- vec_cast.tbl_df.posterior_diff(x, to, ..., x_arg, to_arg)
- vec_cast.posterior_diff.data.frame(x, to, ..., x_arg, to_arg)
- vec_cast.data.frame.posterior_diff(x, to, ..., x_arg, to_arg)

### Parameters
- **x** (object) - The posterior_diff object or input data.
- **y** (object) - The object to compare or cast against.
- **to** (class) - The target class for casting.
- **x_arg** (string) - Argument name for x.
- **y_arg** (string) - Argument name for y.
- **to_arg** (string) - Argument name for to.
```

--------------------------------

### Rank Workflow Results by Metric

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Rank the performance of different workflows within a set based on a specified metric, such as ROC AUC, using `rank_results`. This helps identify the best-performing models.

```R
rank_results(logistic_res, rank_metric = "roc_auc") %>%
  filter(.metric == "roc_auc")
```

--------------------------------

### autoplot.perf_mod_workflow_set

Source: https://tidyposterior.tidymodels.org/reference/autoplot.posterior.html

Produces various types of plots for workflow set objects with computed results, including intervals, posteriors, and ROPE plots.

```APIDOC
## autoplot.perf_mod_workflow_set

### Description
For workflow set objects with computed results, `autoplot()` can produce several types of plots to visualize model performance and posterior distributions.

### Method
S3 method

### Endpoint
N/A (R function)

### Parameters
#### Arguments
- **object** (object) - An object produced by `perf_mod()`, `tidy.perf_mod()`, or a workflow set with computed results.
- **type** (character) - A value of one of: `"intervals"` (for model rank versus posterior probability using interval estimation), `"posteriors"` (density plots for each model), or `"ROPE"` (for practical equivalence probabilities versus workflow rank).
- **prob** (numeric) - A number p (0 < p < 1) indicating the desired probability mass to include in the intervals.
- **size** (numeric) - The size of an effective difference in the units of the chosen metric. For example, a 5 percent increase in accuracy (`size = 0.05`) between two models might be considered a "real" difference.
- **...** (...) - Options passed to `geom_line(stat = "density", ...)`.

### Request Example
```R
data(ex_objects)
autoplot(workflow_set_results, type = "intervals", prob = 0.95)
```

### Response
#### Success Response
- **ggplot2::ggplot()** - A ggplot2::ggplot() object.

#### Response Example
(A ggplot2 object is returned, visualization depends on the input data and the specified `type`)
```

--------------------------------

### perf_mod()

Source: https://tidyposterior.tidymodels.org/reference/index.html

Performs Bayesian analysis of resampling statistics.

```APIDOC
## perf_mod()

### Description
Bayesian Analysis of Resampling Statistics.

### Method
Function Call

### Endpoint
perf_mod(resamples, ...)
```

--------------------------------

### Reexported Objects from Generics

Source: https://tidyposterior.tidymodels.org/reference/reexports.html

This snippet details the 'tidy' generic function reexported from the 'generics' package.

```APIDOC
## Reexport: tidy

### Description
Provides a generic function for tidying model outputs.

### Method
Generic

### Endpoint
N/A

### Parameters
N/A

### Request Body
N/A

### Request Example
N/A

### Response
N/A

### Response Example
N/A
```

--------------------------------

### Posterior Class Compatibility Methods

Source: https://tidyposterior.tidymodels.org/reference/vctrs_methods_posterior.html

These methods extend the 'posterior' class to ensure compatibility with dplyr verbs. Objects with class 'posterior' are defined as tibbles with 'model' (character) and 'posterior' (numeric) columns. Operations that might break these rules will down-cast the object to a basic tibble.

```APIDOC
## Posterior Class Compatibility Methods

### Description
These methods are designed to make the `posterior` class compatible with `dplyr` verbs. A `posterior` object is a tibble with required columns `model` (character) and `posterior` (numeric). If operations break these rules, the object is down-cast to a basic tibble.

### Usage
```R
vec_restore.posterior(x, to, ...)
vec_proxy.posterior(x, ...)
vec_ptype2.posterior.posterior(x, y, ..., x_arg = "", y_arg = "")
vec_ptype2.posterior.tbl_df(x, y, ..., x_arg = "", y_arg = "")
vec_ptype2.tbl_df.posterior(x, y, ..., x_arg = "", y_arg = "")
vec_ptype2.posterior.data.frame(x, y, ..., x_arg = "", y_arg = "")
vec_ptype2.data.frame.posterior(x, y, ..., x_arg = "", y_arg = "")
vec_cast.posterior.posterior(x, to, ..., x_arg = "", to_arg = "")
vec_cast.posterior.tbl_df(x, to, ..., x_arg = "", to_arg = "")
vec_cast.tbl_df.posterior(x, to, ..., x_arg = "", to_arg = "")
vec_cast.posterior.data.frame(x, to, ..., x_arg = "", to_arg = "")
vec_cast.data.frame.posterior(x, to, ..., x_arg = "", to_arg = "")
```

### Source
`R/posteriors-compat.R`
`vctrs_methods_posterior.Rd`
```

--------------------------------

### Fit Bayesian ANOVA Model

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Fits a Bayesian ANOVA model to resampling data using `perf_mod`. Specify the resampling object, a seed for reproducibility, MCMC iterations, number of chains, and refresh rate for output.

```R
rset_mod <- perf_mod(roc_values, seed = 2, iter = 5000, chains = 5, refresh = 0)
```

--------------------------------

### Estimate Model Performance

Source: https://tidyposterior.tidymodels.org/index.html

Use fit_resamples to calculate performance metrics for the defined models across the cross-validation folds.

```R
rs_ctrl <- control_resamples(save_workflow = TRUE)

logistic_reg_glm_res <- 
  logistic_reg_glm_spec %>% 
  fit_resamples(Class ~ ., resamples = folds, control = rs_ctrl)

mars_earth_res <- 
  mars_earth_spec %>% 
  fit_resamples(Class ~ ., resamples = folds, control = rs_ctrl)
```

--------------------------------

### Posterior Diff Class Compatibility Methods

Source: https://tidyposterior.tidymodels.org/reference/vctrs_methods_posterior_diff.html

These methods ensure that objects of class `posterior_diff` can be used with `vctrs` and `dplyr` functions. Operations that would violate the `posterior_diff` structure will down-cast the object to a basic tibble.

```R
vec_restore.posterior_diff(x, to, ...)

vec_proxy.posterior_diff(x, ...)

vec_ptype2.posterior_diff.posterior_diff(x, y, ..., x_arg = "", y_arg = "")

vec_ptype2.posterior_diff.tbl_df(x, y, ..., x_arg = "", y_arg = "")

vec_ptype2.tbl_df.posterior_diff(x, y, ..., x_arg = "", y_arg = "")

vec_ptype2.posterior_diff.data.frame(x, y, ..., x_arg = "", y_arg = "")

vec_ptype2.data.frame.posterior_diff(x, y, ..., x_arg = "", y_arg = "")

vec_cast.posterior_diff.posterior_diff(x, to, ..., x_arg = "", to_arg = "")

vec_cast.posterior_diff.tbl_df(x, to, ..., x_arg = "", to_arg = "")

vec_cast.tbl_df.posterior_diff(x, to, ..., x_arg = "", to_arg = "")

vec_cast.posterior_diff.data.frame(x, to, ..., x_arg = "", to_arg = "")

vec_cast.data.frame.posterior_diff(x, to, ..., x_arg = "", to_arg = "")
```

--------------------------------

### Collect and Filter Performance Metrics

Source: https://tidyposterior.tidymodels.org/articles/Getting_Started.html

Extract performance metrics, such as ROC AUC, from a fitted workflow set using `collect_metrics`. Filter the results to focus on specific metrics for analysis.

```R
collect_metrics(logistic_res) %>%
  filter(.metric == "roc_auc")
```

--------------------------------

### Plotting ROC Differences

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Generates a plot to visualize the difference in ROC statistics between two models. Use this to visually inspect the distribution of differences.

```r
autoplot(preproc_diff) +
  xlab("Difference in ROC (spatial sign - splines)")
```

--------------------------------

### Assessing Normality of Residuals with QQ Plot

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Performs a simple ANOVA on ROC statistics and generates a QQ plot of the residuals to visually assess their consistency with a normal distribution. This serves as a diagnostic for model assumptions.

```r
roc_longer <- 
  roc_values %>%
  select(-splits) %>%
  pivot_longer(cols = c(-id), names_to = "preprocessor", values_to = "roc")

roc_fit <- lm(roc ~ preprocessor, roc_longer)

roc_fit %>%
  augment() %>%
  ggplot(aes(sample = .resid)) + 
  geom_qq() + 
  geom_qq_line(lty = 2)
  coord_fixed(ratio  = 20)
```

--------------------------------

### Plot Bayesian ANOVA Model Results

Source: https://tidyposterior.tidymodels.org/articles/articles/Getting_Started.html

Generates a plot visualizing the results of the Bayesian ANOVA model, likely showing distributions or comparisons of model performance.

```R
autoplot(rset_mod)
```