### Example: Simulate Data and Calculate Mean Outcome

Source: https://stephenrho.github.io/pminternal/reference/boot_optimism.html

This example demonstrates how to simulate data using `sim_dat` and calculate the mean of the outcome variable. It also shows how to remove a linear predictor column.

```R
library(pminternal)
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 1000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.186
dat$LP <- NULL # remove linear predictor
```

--------------------------------

### Install pminternal Package

Source: https://stephenrho.github.io/pminternal/index.html

Install the pminternal package from CRAN or the development version from GitHub.

```R
install.packages("pminternal") # cran
# or 
devtools::install_github("stephenrho/pminternal") # development
```

--------------------------------

### Example: Visualizing Calibration Curves

Source: https://stephenrho.github.io/pminternal/reference/cal_plot.html

This example demonstrates how to use cal_plot after fitting a logistic regression model and validating it. It shows the process of simulating data, fitting a model, and then using 'validate' with 'calib_args' to prepare data for plotting bias-corrected curves.

```R
library(pminternal)
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
# [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ x1 + x2, data=dat, family="binomial")

# to get a plot of bias-corrected calibration we need
# to specify 'eval' argument via 'calib_args'
# this argument specifies at what points to evalulate the
# calibration curve for plotting. The example below uses
# 100 equally spaced points between the min and max
# original prediction.

p <- predict(m1, type="response")
p100 <- seq(min(p), max(p), length.out=100)

m1_iv <- validate(m1, method="cv_optimism", B=10,
                  calib_args = list(eval=p100))
# calib_ags can be used to set other calibration curve
# settings: see pmcalibration::pmcalibration

cal_plot(m1_iv)
```

--------------------------------

### Example of internal validation and summary

Source: https://stephenrho.github.io/pminternal/reference/summary.internal_validate.html

This example demonstrates fitting a logistic regression model and performing internal validation using bootstrap optimism. The `validate` function is used for internal validation, and `summary` is then called on the result to display the validation metrics.

```R
library(pminternal)
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation
summary(m1_iv)
#>           apparent optimism corrected  n
#> C           0.7779  0.00158    0.7764 10
#> Brier       0.1335 -0.00111    0.1346 10
#> Intercept   0.0000 -0.01917    0.0192 10
#> Slope       1.0000  0.00083    0.9992 10
#> Eavg        0.0076  0.00516    0.0024 10
#> E50         0.0064  0.00381    0.0026 10
#> E90         0.0115  0.00882    0.0027 10
#> Emax        0.0580  0.07771   -0.0197 10
#> ECI         0.0110  0.03656   -0.0256 10
```

--------------------------------

### Load Data and Libraries

Source: https://stephenrho.github.io/pminternal/articles/validate-examples.html

Loads necessary libraries (pminternal, Hmisc) and fetches the 'gusto' dataset for use in subsequent examples. It also preprocesses the data by selecting relevant columns and setting the outcome variable.

```R
library(pminternal)
library(Hmisc)

getHdata("gusto")
gusto <- gusto[, c("sex", "age", "hyp", "htn", "hrt", "pmi", "ste", "day30")]

gusto$y <- gusto$day30; gusto$day30 <- NULL

set.seed(234)
gusto <- gusto[sample(1:nrow(gusto), size = 4000),]
```

--------------------------------

### Get Default Calibration Curve Settings

Source: https://stephenrho.github.io/pminternal/reference/cal_defaults.html

Retrieves a list of default settings that can be used when initializing calibration curves with `pmcalibration::pmcalibration`.

```APIDOC
## cal_defaults

### Description
Get default settings for calibration curves

### Usage
```R
cal_defaults(x = NULL)
```

### Arguments
* `x` (ignored) - This argument is currently ignored.

### Value
A list containing default arguments to supply to `pmcalibration::pmcalibration`.
```

--------------------------------

### Get Default Calibration Curve Settings

Source: https://stephenrho.github.io/pminternal/reference/cal_defaults.html

Retrieves a list of default arguments for the `pmcalibration` function. The `x` argument is ignored.

```r
cal_defaults(x = NULL)
```

--------------------------------

### Example: Classification Stability Plot

Source: https://stephenrho.github.io/pminternal/reference/classification_stability.html

Demonstrates how to generate a classification stability plot using simulated data. This example fits a logistic regression model and performs internal validation via bootstrap optimism before plotting.

```R
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation

classification_stability(m1_iv, threshold=.2)
```

--------------------------------

### Run Boot Optimism Analysis

Source: https://stephenrho.github.io/pminternal/reference/boot_optimism.html

Execute the boot_optimism function with specified data, outcome variable, model function, and prediction function. The method is set to 'boot' and B (number of bootstrap replicates) is set to 20 for this example, but should be >= 200 for reliable results.

```R
boot_optimism(data=dat, outcome="y", model_fun=model_fun, pred_fun=pred_fun,
              method="boot", B=20) # B set to 20 for example but should be >= 200
```

--------------------------------

### Example: Plotting calibration stability

Source: https://stephenrho.github.io/pminternal/reference/calibration_stability.html

Demonstrates how to use `calibration_stability` after fitting a logistic regression model and performing bootstrap validation. Ensure B is sufficiently large (>= 200) for practical use.

```R
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)

calibration_stability(m1_iv)
```

--------------------------------

### Example: Plotting Decision Curve Stability

Source: https://stephenrho.github.io/pminternal/reference/dcurve_stability.html

Demonstrates how to simulate data, fit a logistic regression model, perform internal validation using bootstrap optimism, and then plot the decision curve stability.

```R
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation

dcurve_stability(m1_iv)
```

--------------------------------

### Example: Plotting Prediction Stability

Source: https://stephenrho.github.io/pminternal/reference/prediction_stability.html

Demonstrates how to use the prediction_stability function. This involves simulating data, fitting a logistic regression model, performing internal validation via bootstrap optimism, and then plotting the prediction stability.

```R
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation

prediction_stability(m1_iv)
```

--------------------------------

### validate()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Get bias-corrected performance measures via bootstrapping or cross-validation.

```APIDOC
## validate()

### Description
Get bias-corrected performance measures via bootstrapping or cross-validation.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Simulate Data with Missing Predictors

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Generates synthetic data with multiple predictors, some of which contain missing values. This setup is useful for testing methods that handle missing data.

```R
library(pminternal)
library(mice)
#> 
#> Attaching package: 'mice'
#> The following object is masked from 'package:stats':
#> 
#>     filter
#> The following objects are masked from 'package:base':
#> 
#>     cbind, rbind

# make some data
set.seed(2345)
n <- 800
p <- 5

X <- matrix(rnorm(n*p), nrow = n, ncol = p)
LP <- -1 + X %*% c(-1, 1, -.5, .5, 0)
y <- rbinom(n, 1, plogis(LP))

mean(y)
#> [1] 0.33125

datcomp <- data.frame(y, X)

# add missingness
datmis <- datcomp
datmis$X1[sample(n, 200)] <- NA
datmis$X2[sample(n, 200)] <- NA
datmis$X3[sample(n, 200)] <- NA
```

--------------------------------

### Internal Validation using rms package and pminternal::validate

Source: https://stephenrho.github.io/pminternal/reference/validate.html

Shows how to use the pminternal::validate function with a model fitted using the rms package's lrm function. Similar to the previous example, B should be >= 200 for robust validation.

```R
library(rms)
#> Loading required package: Hmisc
#> 
#> Attaching package: ‘Hmisc’
#> The following objects are masked from ‘package:base’: 
#> 
#>     format.pval, units
#> 
#> Attaching package: ‘rms’
#> The following object is masked from ‘package:pminternal’: 
#> 
#>     validate
m2 <- lrm(y ~ ., data=dat)
m2_iv <- pminternal::validate(m2, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation
```

--------------------------------

### print.internal_boot

Source: https://stephenrho.github.io/pminternal/reference/print.internal_boot.html

Prints an internal_boot object to the console, displaying estimates with a specified number of digits.

```APIDOC
## print.internal_boot

### Description
Prints an internal_boot object to the console, displaying estimates with a specified number of digits.

### Method
S3 method for class 'internal_boot'

### Parameters
#### Arguments
- **x** (object) - Required - an object created with `boot_optimism`
- **digits** (number) - Optional - number of digits to print (default = 2)
- **...** (any) - Optional - additional arguments to print

### Value
Invisibly returns `x` and prints estimates to console.
```

--------------------------------

### boot_optimism()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Calculate optimism and bias-corrected scores via bootstrap resampling.

```APIDOC
## boot_optimism()

### Description
Calculate optimism and bias-corrected scores via bootstrap resampling.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Fit a GLM model and perform bootstrap validation

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Loads data, prepares it, fits a binomial GLM model, and then uses the `validate` function from the `pminternal` package to perform bootstrap validation and estimate optimism-corrected performance metrics. It's recommended to use B >= 200 for more reliable bootstrap validation.

```R
library(pminternal)
library(Hmisc)

getHdata("gusto")
gusto <- gusto[, c("sex", "age", "hyp", "htn", "hrt", "pmi", "ste", "day30")]

gusto$y <- gusto$day30; gusto$day30 <- NULL
mean(gusto$y) # outcome rate

set.seed(234)
gusto <- gusto[sample(1:nrow(gusto), size = 4000),]

mod <- glm(y ~ ., data = gusto, family = "binomial")

mod_iv <- validate(mod, B = 20)

```

--------------------------------

### Summarize internal_validate object

Source: https://stephenrho.github.io/pminternal/reference/summary.internal_validate.html

Use this function to get a data.frame summary of internal validation results. The summary includes apparent score, optimism, bias-corrected score, and number of successful resamples/folds. It can also include confidence intervals if added via `confint.internal_validate`.

```R
summary(object, ignore_scores = "^cal_plot", ...)
```

--------------------------------

### Print internal_boot object

Source: https://stephenrho.github.io/pminternal/reference/print.internal_boot.html

This is the S3 method for printing internal_boot objects. It controls how estimates are displayed to the console. Use this when you want to view the results of a bootstrap optimism calculation.

```R
print(x, digits = 2, ...)
```

--------------------------------

### Validate Models from Different Packages (gbm, mgcv, rms)

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Demonstrates internal validation for models created with `gbm`, `mgcv`, and `rms` packages. Note that `rms::validate` is used to avoid conflicts.

```R
library(gbm)
# syntax y ~ . does not work with gbm
mod <- gbm(y ~ sex + age + hyp + htn + hrt + pmi + ste, 
           data = gusto, distribution = "bernoulli", interaction.depth = 2)

(gbm_iv <- validate(mod, B = 20))
```

```R
library(mgcv)

mod <- gam(y ~ sex + s(age) + hyp + htn + hrt + pmi + ste, 
           data = gusto, family = "binomial")

(gam_iv <- validate(mod, B = 20))
```

```R
mod <- rms::lrm(y ~ ., data = gusto) 
# not loading rms to avoid conflict with rms::validate...

(lrm_iv <- validate(mod, B = 20))
```

--------------------------------

### summary(_<internal_validate>_)

Source: https://stephenrho.github.io/pminternal/reference/index.html

Summarize a internal_validate object.

```APIDOC
## summary(_<internal_validate>_)

### Description
Summarize a internal_validate object.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### calibration_stability()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Plot calibration stability across bootstrap replicates.

```APIDOC
## calibration_stability()

### Description
Plot calibration stability across bootstrap replicates.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Validate Model Performance

Source: https://stephenrho.github.io/pminternal/index.html

Performs bootstrap validation on a model using specified functions for fitting and prediction. The 'boot_optimism' method is used here.

```R
(val <- validate(data = dat, outcome = "y", 
                 model_fun = lasso_fun, pred_fun = lasso_predict, 
                 method = "boot_optimism", B = 100))
```

--------------------------------

### print.internal_validate

Source: https://stephenrho.github.io/pminternal/reference/print.internal_validate.html

Prints a summary of an internal_validate object.

```APIDOC
## print.internal_validate

### Description
Prints a summary of an `internal_validate` object.

### Usage
```R
# S3 method for class 'internal_validate'
print(x, digits = 2, ...)
```

### Arguments

* `x` (internal_validate): The `internal_validate` object to print.
* `digits` (numeric): The number of digits to print. Defaults to 2.
* `...`: Optional arguments passed to the print function.
```

--------------------------------

### Test Lasso Functions and Validate Model

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Tests the custom `lasso_fun` and `lasso_predict` functions with sample data and then uses them within the `validate` function to assess model performance using cross-validation for optimism estimation. The `calib_args` are used to specify evaluation points for the calibration plot.

```R
lasso_app <- lasso_fun(gusto)
lasso_p <- lasso_predict(model = lasso_app, data = gusto)
```

```R
# for calibration plot
eval <- seq(min(lasso_p), max(lasso_p), length.out=100)

iv_lasso <- validate(method = "cv_optimism", data = gusto, 
                     outcome = "y", model_fun = lasso_fun, 
                     pred_fun = lasso_predict, B = 10, 
                     calib_args=list(eval=eval))

iv_lasso
```

```R
cal_plot(iv_lasso)
```

--------------------------------

### Internal Validation of Logistic Regression Model using Bootstrap Optimism

Source: https://stephenrho.github.io/pminternal/reference/validate.html

Demonstrates how to perform internal validation of a logistic regression model using the bootstrap optimism method with the pminternal package. It's recommended to use B >= 200 for practical applications.

```R
library(pminternal)
set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
#> [1] 0.1985
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)
#> It is recommended that B >= 200 for bootstrap validation
m1_iv
#>
#>           apparent optimism corrected  n
#> C           0.7779  0.00158    0.7764 10
#> Brier       0.1335 -0.00111    0.1346 10
#> Intercept   0.0000 -0.01917    0.0192 10
#> Slope       1.0000  0.00083    0.9992 10
#> Eavg        0.0076  0.00516    0.0024 10
#> E50         0.0064  0.00381    0.0026 10
#> E90         0.0115  0.00882    0.0027 10
#> Emax        0.0580  0.07771   -0.0197 10
#> ECI         0.0110  0.03656   -0.0256 10
```

--------------------------------

### Validate Model and Generate Calibration Plot

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Performs bootstrap validation on a model and generates a calibration plot. The plot can be customized with confidence intervals and bias-corrected colors.

```R
p <- predict(mod, type="response")

p_range <- seq(min(p), max(p), length.out=100)

mod_iv2 <- validate(mod, B = 20, 
                    calib_args = list(
                      eval=p_range, 
                      smooth="rcs", 
                      nk=5)
                    )

calp <- cal_plot(mod_iv2)
```

```R
library(ggplot2)

ggplot(calp, aes(x=predicted)) + 
  geom_abline(lty=2) +
  geom_line(aes(y=apparent, color="Apparent")) +
  geom_line(aes(y=bias_corrected, color="Bias-Corrected")) +
  geom_histogram(data = data.frame(p = p), aes(x=p, y=after_stat(density)*.01),
                 binwidth = .001, inherit.aes = F, alpha=1/2) +
  labs(x="Predicted Risk", y="Estimated Risk", color=NULL)
```

```R
cal_plot(mod_iv2, bc_col = "red")
```

--------------------------------

### Fit GLM with LASSO Penalty

Source: https://stephenrho.github.io/pminternal/index.html

Fits a LASSO-penalized binomial GLM using cross-validation to determine the optimal lambda. Requires the 'glmnet' package.

```R
library(glmnet)
lasso_fun <- function(data, ...){ y <- data$y
  x <- as.matrix(data[, which(colnames(data) != "y")])
  
  cv <- cv.glmnet(x=x, y=y, alpha=1, nfolds = 10, family="binomial")
  lambda <- cv$lambda.min
  
  glmnet(x=x, y=y, alpha = 1, lambda = lambda, family="binomial")
}
```

--------------------------------

### Flexible Lognet Model Function with Alpha Parameter

Source: https://stephenrho.github.io/pminternal/articles/validate-examples.html

Creates a flexible model function 'lognet_fun' that can handle both lasso (L1) and ridge (L2) penalization by accepting an 'alpha' argument. If 'alpha' is not provided, it defaults to 0 (ridge). This function is designed to be used with the validate function.

```R
lognet_fun <- function(data, ...){
  
  dots <- list(...)
  if ("alpha" %in% names(dots)){
    alpha <- dots[["alpha"]]
  } else{
    alpha <- 0
  }

  y <- data$y
  x <- data[, c('sex', 'age', 'hyp', 'htn', 'hrt', 'pmi', 'ste')]
  x$sex <- as.numeric(x$sex == "male")
  x$pmi <- as.numeric(x$pmi == "yes")
  x <- as.matrix(x)
  
  cv <- glmnet::cv.glmnet(x=x, y=y, alpha = alpha, nfolds = 10, family="binomial")
  lambda <- cv$lambda.min
  
  glmnet::glmnet(x=x, y=y, alpha = alpha, lambda = lambda, family="binomial")
}

validate(method = "cv_optimism", data = gusto, 
         outcome = "y", model_fun = lognet_fun, 
         pred_fun = ridge_predict, B = 10, alpha = 0.5)
```

--------------------------------

### Identify Missing Data Patterns (R)

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Uses the `md.pattern` function from the `mice` package to identify and visualize the patterns of missing data within a dataset.

```r
md <- md.pattern(datmis)

```

--------------------------------

### S3 method for class 'internal_validate'

Source: https://stephenrho.github.io/pminternal/reference/print.internal_validate.html

This is the S3 method signature for printing an object of class 'internal_validate'. It takes the object 'x' and optional 'digits' and other arguments.

```R
# S3 method for class 'internal_validate'
print(x, digits = 2, ...)
```

--------------------------------

### Validation of Stacked Multiple Imputation Model (R)

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Demonstrates how to validate a stacked multiple imputation model using the `validate` function. It shows the apparent, optimism, and corrected performance metrics for the model.

```r
(mi_val <- validate(data = datmis, 
                    model_fun = model_mi, pred_fun = pred_mi, 
                    outcome = "y", method = "cv_o"))
#>           apparent optimism corrected  n
#> C            0.821   0.0020     0.819 10
#> Brier        0.160  -0.0007     0.161 10
#> Intercept    0.221   0.0058     0.216 10
#> Slope        1.267  -0.0446     1.312 10
#> Eavg         0.038  -0.0330     0.071 10
#> E50          0.034  -0.0202     0.054 10
#> E90          0.079  -0.0591     0.138 10
#> Emax         0.084  -0.1489     0.232 10
#> ECI          0.200  -0.9423     1.142 10

```

--------------------------------

### print.internal_cv

Source: https://stephenrho.github.io/pminternal/reference/print.internal_cv.html

Prints an internal_cv object to the console, showing cross-validation estimates.

```APIDOC
## print.internal_cv

### Description
Prints an internal_cv object to the console, showing cross-validation estimates.

### Usage
```R
# S3 method for class 'internal_cv'
print(x, digits = 2, ...)
```

### Arguments

*   `x` (object) - An object created with `crossval`.
*   `digits` (numeric) - Number of digits to print (default = 2).
*   `...` - Additional arguments to print.

### Value
Invisibly returns `x` and prints estimates to the console.
```

--------------------------------

### print.internal_validatesummary

Source: https://stephenrho.github.io/pminternal/reference/print.internal_validatesummary.html

Prints a summary of an internal_validate object. This is an S3 method for the internal_validatesummary class.

```APIDOC
## print.internal_validatesummary

### Description
Prints a summary of an internal_validate object.

### Usage
```R
print(x, digits = 2, ...)
```

### Arguments

*   `x` (internal_validatesummary): An object of class `internal_validatesummary`.
*   `digits` (numeric): The number of digits to print. Defaults to 2.
*   `...` (any): Ignored arguments.

### Value
Returns the input object `x` invisibly after printing the summary.
```

--------------------------------

### score_binary()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Score predictions for binary events.

```APIDOC
## score_binary()

### Description
Score predictions for binary events.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### confint(_<internal_validate>_)

Source: https://stephenrho.github.io/pminternal/reference/index.html

Confidence intervals for bias-corrected performance measures.

```APIDOC
## confint(_<internal_validate>_)

### Description
Confidence intervals for bias-corrected performance measures.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### S3 Method for internal_validate Class

Source: https://stephenrho.github.io/pminternal/reference/confint.internal_validatesummary.html

This is the signature for the `confint` S3 method applied to objects of class `internal_validate`. It is used to compute confidence intervals for performance measures.

```R
# S3 method for class 'internal_validate'
confint(
  object,
  parm,
  level = 0.95,
  method = c("shifted", "twostage"),
  ci_type = c("perc", "norm"),
  R = 1000,
  add = TRUE,
  ...
)
```

--------------------------------

### cal_plot()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Plot apparent and bias-corrected calibration curves.

```APIDOC
## cal_plot()

### Description
Plot apparent and bias-corrected calibration curves.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Generate a calibration stability plot

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Generates a calibration stability plot using the results from a `validate` call. This function uses default arguments for calibration curve estimation, which can be inspected via `pminternal:::cal_defaults()`.

```R
calibration_stability(mod_iv)
```

--------------------------------

### confint.internal_validate S3 Method

Source: https://stephenrho.github.io/pminternal/reference/confint.internal_validatesummary.html

Calculates confidence intervals for bias-corrected performance measures for objects created by the `validate` function.

```APIDOC
## confint.internal_validate

### Description
Calculates confidence intervals for bias-corrected performance measures for objects created by the `validate` function.

### Usage
```R
confint(object, parm, level = 0.95, method = c("shifted", "twostage"), ci_type = c("perc", "norm"), R = 1000, add = TRUE, ...)
```

### Arguments
* `object`: An object created by a call to `validate`.
* `parm`: A specification of which performance measures to include, either by number or name. If missing, all measures are used.
* `level`: The desired confidence level (default is 0.95).
* `method`: The method for calculating CIs, either "shifted" or "twostage".
* `ci_type`: The type of bootstrap CIs: "perc" (percentile) or "norm" (normal approximation).
* `R`: The number of bootstrap replicates (default is 1000).
* `add`: If TRUE, adds the CIs to the object; otherwise, returns only the CIs.
* `...`: Additional arguments (currently ignored).

### Value
A list containing two matrices: one for apparent performance CIs and one for bias-corrected performance CIs. Each matrix has columns for the lower and upper confidence limits, labeled with the corresponding percentage (e.g., 2.5% and 97.5%).

### Details
* **shifted**: Shifts bootstrap CIs for apparent performance by the optimism value. This is faster as it only requires calculating apparent performance for each replicate.
* **twostage**: Runs the entire validation procedure on each bootstrap resample. This method is computationally intensive as it involves R * B replicates (where B is the number of inner replicates from the original `validate` call).

### References
Noma, H., Shinozaki, T., Iba, K., Teramukai, S., & Furukawa, T. A. (2021). Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap‐based optimism correction methods. Statistics in Medicine, 40(26), 5691-5701.
```

--------------------------------

### Plotting Calibration Curves

Source: https://stephenrho.github.io/pminternal/reference/cal_plot.html

Use cal_plot to visualize apparent and bias-corrected calibration curves. Ensure the input object 'x' is from a 'validate' call with 'eval' specified. Customize plot appearance with arguments like colors and line types.

```R
cal_plot(
  x,
  xlim,
  ylim,
  xlab,
  ylab,
  app_col,
  bc_col,
  app_lty,
  bc_lty,
  plotci = c("if", "yes", "no")
)
```

--------------------------------

### prediction_stability()

Source: https://stephenrho.github.io/pminternal/reference/index.html

Plot prediction stability across bootstrap replicates.

```APIDOC
## prediction_stability()

### Description
Plot prediction stability across bootstrap replicates.

### Method
N/A (Function Signature)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A
```

--------------------------------

### Identify Missing Data Patterns

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Generates a pattern string for each row based on the presence (1) or absence (0) of missing values in specified columns. Useful for categorizing missing data.

```R
datmis$mdp <- apply(datmis[, paste0("X", 1:5)], 1, \(x) paste0(as.numeric(is.na(x)), collapse = ""))

table(datmis$mdp)
```

--------------------------------

### Calculate Bootstrap Confidence Intervals for Model Validation

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

Calculates bootstrap confidence intervals for model validation metrics. This is useful for assessing the uncertainty of the validation results.

```R
(mod_iv2 <- confint(mod_iv2, method = "shifted", R=100))
```

--------------------------------

### Assess Model Performance Before Missingness

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Fits a logistic regression model to complete data and evaluates its performance using cross-validation. This serves as a baseline before introducing missing data.

```R
# assess performance before introducing missingness
mod <- glm(y ~ ., data = datcomp, family = "binomial")
(comp_val <- validate(mod, method = "cv_o"))
#>           apparent optimism corrected  n
#> C         0.801121  -0.0013     0.802 10
#> Brier     0.164447   0.0000     0.164 10
#> Intercept 0.000000  -0.0320     0.032 10
#> Slope     1.000000  -0.0672     1.067 10
#> Eavg      0.000004  -0.0476     0.048 10
#> E50       0.000004  -0.0421     0.042 10
#> E90       0.000005  -0.0949     0.095 10
#> Emax      0.000009  -0.1058     0.106 10
#> ECI       0.000000  -0.4810     0.481 10
```

--------------------------------

### Validate Models with Missing Data Handling

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Applies a validation procedure (e.g., cross-validation) to assess the performance of models fitted using the `model_ps` function, which accounts for missing data patterns. Requires `validate` function from a modeling package.

```R
(sub_val <- validate(data = datmis, 
                    model_fun = model_ps, pred_fun = pred_ps, 
                    submodels = submodels, 
                    outcome = "y", method = "cv_o"))
```

--------------------------------

### Print internal_validate summary

Source: https://stephenrho.github.io/pminternal/reference/print.internal_validatesummary.html

This is an S3 method for the internal_validate object class. It prints a summary of the validation results, controlling the number of digits displayed.

```R
# S3 method for class 'internal_validatesummary'
print(x, digits = 2, ...)
```

--------------------------------

### Define custom sensitivity and specificity function

Source: https://stephenrho.github.io/pminternal/articles/pminternal.html

This R function calculates sensitivity and specificity. It accepts an optional `threshold` argument, defaulting to 0.5, to determine the predicted class. Ensure the input `y` is binary (1/0 or logical).

```r
sens_spec <- function(y, p, ...){ 
  # this function supports an optional
  # arg: threshold (set to .5 if not specified)
  dots <- list(...)
  if ("threshold" %in% names(dots)){
    thresh <- dots[["threshold"]]
  } else{
    thresh <- .5
  }
  # make sure y is 1/0
  if (is.logical(y)) y <- as.numeric(y)
  # predicted 'class'
  pcla <- as.numeric(p > thresh) 

  sens <- sum(y==1 & pcla==1)/sum(y==1)
  spec <- sum(y==0 & pcla==0)/sum(y==0)

  scores <- c(sens, spec)
  names(scores) <- c("Sensitivity", "Specificity")
  
  return(scores)
}
```

--------------------------------

### Stacked Multiple Imputation Model Functions (R)

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Defines functions for training a model using multiple imputation and for predicting with stacked imputation. The `model_mi` function pools coefficients, and `pred_mi` handles imputation of new data while ignoring it for model fitting but including it for imputation.

```r
model_mi <- function(data, ...){
  imp <- mice::mice(data, printFlag = FALSE) # 5 imputed datasets via pmm
  fits <- with(imp, glm(y ~ X1 + X2 + X3 + X4 + X5, family=binomial))
  B <- mice::pool(fits)$pooled$estimate # pooled coefs for predictions
  # save pooled coefficients and the data to do stacked imputation at validation
  list(B, data) 
}
```

```r
pred_mi <- function(model, data, ...){
  # extract pooled coefs
  B <- model[[1]]
  # stack the new data on the model fit data (model[[2]])
  dstacked <- rbind(model[[2]], data)
  i <- (nrow(model[[2]]) + 1):(nrow(dstacked))
  # impute stacked data
  # use ignore argument to ensure the test data doesn't influence
  # the imputation model but still gets imputed
  imp <- mice::mice(dstacked, printFlag = FALSE, 
                    ignore = seq(nrow(dstacked)) %in% i)
  # get logit predicted risks for each imputed data set
  preds <- sapply(seq(imp$m), \(x){
    # complete(imp, x)[i, ] = get complete data set x and extract the 
    # validation data i
    X <- model.matrix(~ X1 + X2 + X3 + X4 + X5, data = mice::complete(imp, x)[i, ]) 
    X %*% B
  })
  # average logit predictions, transform invlogit, and return
  plogis(apply(preds, 1, mean)) 
}
```

--------------------------------

### Define Submodels for Missing Data Patterns

Source: https://stephenrho.github.io/pminternal/articles/missing-data.html

Defines a list of regression formulas (submodels) for different missing data patterns. Each formula specifies the relationship between the outcome variable 'y' and predictors, tailored to patterns identified by 'mdp'.

```R
submodels <- list(
  "00000" = y ~ X1 + X2 + X3 + X4 + X5,
  "00100" = y ~ X1 + X2 + X4 + X5,
  "01000" = y ~ X1 + X3 + X4 + X5,
  "01100" = y ~ X1 + X4 + X5,
  "10000" = y ~ X2 + X3 + X4 + X5,
  "10100" = y ~ X2 + X4 + X5,
  "11000" = y ~ X3 + X4 + X5,
  "11100" = y ~ X4 + X5
)
```