### Install nestedcv from CRAN Source: https://github.com/myles-lewis/nestedcv/blob/main/README.md Installs the nestedcv package from the Comprehensive R Archive Network (CRAN). This is the standard method for installing R packages. ```r install.packages("nestedcv") ``` -------------------------------- ### Install nestedcv from GitHub Source: https://github.com/myles-lewis/nestedcv/blob/main/README.md Installs the nestedcv package directly from its GitHub repository using the devtools package. This is useful for installing the latest development version. ```r devtools::install_github("myles-lewis/nestedcv") ``` -------------------------------- ### SMOTE for Class Imbalance Handling Source: https://context7.com/myles-lewis/nestedcv/llms.txt Illustrates the SMOTE (Synthetic Minority Oversampling Technique) algorithm for balancing imbalanced datasets. It generates synthetic samples for the minority class to match the majority class, improving model performance on imbalanced data. ```r library(nestedcv) # Create imbalanced dataset set.seed(123) x <- matrix(rnorm(150 * 100), 150, 100) y <- factor(rbinom(150, 1, 0.2)) # 20% minority class table(y) # 0 1 # 120 30 # Apply SMOTE to balance classes out <- smote(y, x, k = 5) # k nearest neighbors y_balanced <- out$y x_balanced <- out$x table(y_balanced) # 0 1 # 120 120 # Specify amount of oversampling out <- smote(y, x, over = 2) # double minority samples table(out$y) # Use within nested CV (recommended approach) fit <- nestcv.glmnet( y = y, x = x, family = "binomial", balance = "smote", balance_options = list(k = 5), cv.cores = 2 ) ``` -------------------------------- ### Extended Performance Metrics with metrics() Source: https://context7.com/myles-lewis/nestedcv/llms.txt Shows how to use the `metrics` function to obtain comprehensive performance metrics for classification models, including AUC, Accuracy, Balanced Accuracy, PR-AUC, Kappa, F1, and MCC. It can also display inner CV metrics for comparison. ```r library(nestedcv) library(mlbench) data(Sonar) y <- Sonar$Class x <- Sonar[, -61] fit <- nestcv.glmnet( y = y, x = x, family = "binomial", alphaSet = 1, n_outer_folds = 5, cv.cores = 2 ) # Basic metrics metrics(fit) # AUC Accuracy Balanced.Accuracy nvar # 0.85 0.78 0.78 12 # Extended metrics metrics(fit, extra = TRUE) # AUC Accuracy Balanced.Accuracy PR.AUC Kappa F1 MCC nvar # 0.85 0.78 0.78 0.82 0.56 0.77 0.56 12 # Inner CV metrics for comparison (shows optimistic bias) metrics(fit, innerCV = TRUE) ``` -------------------------------- ### Random Sample for Class Balancing Source: https://context7.com/myles-lewis/nestedcv/llms.txt Demonstrates `randomsample` for class balancing via random oversampling of the minority class or undersampling of the majority class. It can be used standalone or integrated within nested cross-validation. ```r library(nestedcv) # Imbalanced data set.seed(123) x <- matrix(rnorm(150 * 100), 150, 100) y <- factor(rbinom(150, 1, 0.2)) table(y) # 0 1 # 120 30 # Oversample minority to match majority out <- randomsample(y, x) table(out$y) # 0 1 # 120 120 # Combined over and undersampling out <- randomsample(y, x, minor = 2, major = 0.5) # minor=2: double minority samples # major=0.5: keep 50% of majority samples table(out$y) # Nested CV with balanced sampling fit <- nestcv.glmnet( y = y, x = x, family = "binomial", balance = "randomsample", balance_options = list(minor = 1, major = 0.5), filterFUN = ttest_filter, cv.cores = 2 ) # Check class balance in CV folds class_balance(fit) ``` -------------------------------- ### ReliefF Algorithm for Feature Selection Source: https://context7.com/myles-lewis/nestedcv/llms.txt Applies the ReliefF algorithm and related methods from the CORElearn package for feature selection. It requires outcome (y), predictors (x), and the number of features to select. Different estimators can be specified. ```r library(nestedcv) data(iris) y <- iris$Species x <- iris[, 1:4] # ReliefF filter filt <- relieff_filter( y = y, x = x, nfilter = 3, estimator = "ReliefFequalK" ) # Full importance scores relieff_filter(y, x, type = "full") # Alternative estimators available from CORElearn filt <- relieff_filter(y, x, nfilter = 3, estimator = "ReliefFexpRank") ``` -------------------------------- ### Perform Repeated Nested Cross-Validation using glmnet Source: https://context7.com/myles-lewis/nestedcv/llms.txt Demonstrates how to perform repeated nested cross-validation using the `nestcv.glmnet` function and the `repeatcv` pipe for robust performance estimates. It includes setting random seeds for reproducibility and viewing summary statistics. ```r library(nestedcv) library(mlbench) data(Sonar) y <- Sonar$Class x <- Sonar[, -61] # Single nested CV fit <- nestcv.glmnet(y, x, family = "binomial", alphaSet = 1, n_outer_folds = 5, cv.cores = 2) # Repeated nested CV using pipe set.seed(123, "L'Ecuyer-CMRG") # for reproducible parallel RNG repcv <- nestcv.glmnet(y, x, family = "binomial", alphaSet = 1, n_outer_folds = 5) |> repeatcv(n = 10, rep.cores = 4) # View results across repeats repcv # Summary statistics summary(repcv) # Fixed folds for fair model comparison folds <- repeatfolds(y, repeats = 10, n_outer_folds = 5) repcv1 <- nestcv.glmnet(y, x, family = "binomial", alphaSet = 1, n_outer_folds = 5) |> repeatcv(10, repeat_folds = folds, rep.cores = 4) repcv2 <- nestcv.train(y, x, method = "rf", n_outer_folds = 5) |> repeatcv(10, repeat_folds = folds, rep.cores = 4) # Compare models using same folds summary(repcv1) summary(repcv2) # Plot smoothed ROC from repeated CV plot(repcv$roc, main = "ROC from Repeated Nested CV") ``` -------------------------------- ### Precision-Recall Curves with prc() Source: https://context7.com/myles-lewis/nestedcv/llms.txt Demonstrates how to generate and plot Precision-Recall (PR) curves using the `prc` function for binary classification models. It also shows how to extract the PR AUC value and compare it visually with the ROC curve. ```r library(nestedcv) library(mlbench) data(Sonar) y <- Sonar$Class x <- Sonar[, -61] fit <- nestcv.glmnet(y, x, family = "binomial", cv.cores = 2) # Generate PR curve pr <- prc(fit) # PR AUC value pr$auc # Plot PR curve plot(pr, main = "Precision-Recall Curve") # Compare with ROC par(mfrow = c(1, 2)) plot(fit$roc, main = "ROC Curve") plot(pr, main = "PR Curve") ``` -------------------------------- ### Visualize Model Tuning and Results with nestedcv Plotting Functions Source: https://context7.com/myles-lewis/nestedcv/llms.txt Illustrates various plotting functions from the nestedcv package for visualizing model tuning parameters (alpha, lambda), variable importance, expression levels, and ROC curves. These plots aid in understanding model behavior and performance. ```r library(nestedcv) fit <- nestcv.glmnet(y, x, family = "binomial", alphaSet = seq(0.5, 1, 0.1), cv.cores = 2) # Plot alpha tuning across outer folds plot_alphas(fit) # Plot lambda tuning across outer folds plot_lambdas(fit, showLegend = "bottomright") # Plot cva.glmnet object from specific fold plot(fit$outer_result[[1]]$cvafit) # alpha vs deviance plot(fit$outer_result[[1]]$cvafit, xaxis = "lambda") # lambda curves plot(fit$outer_result[[1]]$cvafit, xaxis = "nvar") # num variables # Variable importance plot plot_varImp(fit, abs = TRUE, size = TRUE) # Boxplot of predictor expression levels boxplot_expression(fit) # ROC curve plot(fit$roc) library(pROC) legend("bottomright", legend = paste("AUC =", round(auc(fit$roc), 3)), bty = "n") ``` -------------------------------- ### Filter with glmnet_filter using Non-Zero Coefficients Count Source: https://context7.com/myles-lewis/nestedcv/llms.txt Illustrates feature filtering with glmnet_filter using the 'nonzero' method, which counts the number of non-zero coefficients for each feature across the lambda path. This method helps identify features that are selected at any regularization strength. ```r filt <- glmnet_filter( y = y, x = x, family = "binomial", nfilter = 50, method = "nonzero" ) ``` -------------------------------- ### Filter with glmnet_filter and Forced Variables Source: https://context7.com/myles-lewis/nestedcv/llms.txt Shows how to use glmnet_filter while forcing specific variables to be included in the model without shrinkage. The `force_vars` argument ensures these variables are always selected, regardless of their regularization path. ```r filt <- glmnet_filter( y = y, x = x, family = "binomial", force_vars = c("V1", "V2"), nfilter = 50 ) ``` -------------------------------- ### Plotting Nested CV Results for glmnet Source: https://github.com/myles-lewis/nestedcv/blob/main/README.md Generates plots to visualize the results of nested cross-validation performed with glmnet. Includes plots for alpha and lambda tuning, and detailed plots for individual outer CV folds. ```r plot_alphas(res) plot_lambdas(res) # Plot for a specific outer CV fold plot(res$outer_result[[1]]$cvafit) ``` -------------------------------- ### Universal Statistical Filter with stat_filter Source: https://context7.com/myles-lewis/nestedcv/llms.txt Demonstrates the use of `stat_filter` for feature selection with mixed data types. It calculates appropriate statistical tests (correlation, t-test, ANOVA) based on predictor and outcome types, and can return full statistics or top predictors. ```r library(nestedcv) library(mlbench) data(BostonHousing2) dat <- BostonHousing2 y <- dat$cmedv # continuous outcome x <- subset(dat, select = -c(cmedv, medv, town)) # Full statistics for all predictors stat_filter(y, x, type = "full") # Returns appropriate test statistic based on data type: # - Correlation for continuous predictor vs continuous outcome # - t-test for binary predictor vs continuous outcome # - ANOVA for categorical predictor vs continuous outcome # Get top predictors filt <- stat_filter(y, x, nfilter = 5) ``` -------------------------------- ### Filter with glmnet_filter using Mean Absolute Coefficients Source: https://context7.com/myles-lewis/nestedcv/llms.txt Demonstrates filtering features using the glmnet_filter function with the 'mean' method to consider mean absolute coefficients across the lambda path. This is useful for selecting features that have consistent importance across different regularization strengths. ```r filt <- glmnet_filter( y = y, x = x, family = "binomial", nfilter = 50, alpha = 0.5, # elastic net mixing method = "mean" # mean absolute coefficients ) ``` -------------------------------- ### Nested CV with caret and ranger Source: https://github.com/myles-lewis/nestedcv/blob/main/README.md Performs nested cross-validation using the caret framework with the ranger package for model training. This allows leveraging caret's extensive model options within a nested CV structure. ```r library(nestedcv) # Assuming y and x are already defined from previous example # y <- iris$Species # x <- as.matrix(iris[, -5]) # cores <- parallel::detectCores(logical = FALSE) res <- nestcv.train(y, x, method = "ranger", cv.cores = cores) summary(res) ``` -------------------------------- ### Outer Cross-Validation with Linear Model (Formula Interface) Source: https://context7.com/myles-lewis/nestedcv/llms.txt Fits a linear model using outer cross-validation with a formula interface. It takes a formula, data frame, model type ('lm'), and number of outer folds. The summary provides RMSE, MAE, and R-squared. ```r y_cont <- -3 + 0.5 * x$marker1 + 2 * x$marker2 + rnorm(nrow(x), 0, 2) df <- cbind(x, outcome = y_cont) fit_lm <- outercv( outcome ~ ., data = df, model = "lm", n_outer_folds = 10 ) summary(fit_lm) # Returns RMSE, MAE, R-squared ``` -------------------------------- ### Simple Outer CV for Fixed Parameter Models in R Source: https://context7.com/myles-lewis/nestedcv/llms.txt A convenience function for performing single-loop cross-validation on models with fixed hyperparameters. This is useful for quick evaluation of models where hyperparameter tuning is not required or has already been performed. It simplifies the process of obtaining performance estimates for such models. ```r library(nestedcv) library(randomForest) # Sigmoid function for generating binary outcome sigmoid <- function(x) { 1 / (1 + exp(-x)) } # Prepare data data(iris) x <- iris[, 1:4] colnames(x) <- c("marker1", "marker2", "marker3", "marker4") x <- as.data.frame(apply(x, 2, scale)) y <- sigmoid(0.5 * x$marker1 + 2 * x$marker2) > runif(nrow(x)) y <- factor(y) # Example usage (assuming outercv function exists and is used like this): # fit_outer <- outercv( # y = y, # x = x, # model_function = randomForest, # or another model with fixed params # n_folds = 10, # # ... other arguments specific to the model or evaluation # ) # summary(fit_outer) ``` -------------------------------- ### Elastic Net Sparsity Feature Selection Filter Source: https://context7.com/myles-lewis/nestedcv/llms.txt Selects features based on the sparsity induced by elastic net regression. This function requires predictor matrix (x) and outcome vector (y). It's particularly useful for high-dimensional datasets. ```r library(nestedcv) # Generate data set.seed(123) x <- matrix(rnorm(100 * 500), 100, 500) y <- factor(rbinom(100, 1, 0.5)) # Example usage (assuming glmnet_filter function exists and is loaded) # filt <- glmnet_filter(y, x, nfilter = 10, alpha = 0.5) # alpha is the mixing parameter ``` -------------------------------- ### Random Forest Feature Selection Filters (rf_filter, ranger_filter) Source: https://context7.com/myles-lewis/nestedcv/llms.txt Utilizes variable importance from Random Forest models for feature selection. `rf_filter` uses the `randomForest` package, while `ranger_filter` uses the faster `ranger` implementation. Both require outcome (y), predictors (x), and the number of features to select. ```r library(nestedcv) data(iris) y <- iris$Species x <- iris[, 1:4] # Random forest filter using randomForest package filt <- rf_filter( y = y, x = x, nfilter = 3, ntree = 500, mtry = 2 ) # Get full variable importance scores vi <- rf_filter(y, x, type = "full") vi # Petal.Width Petal.Length Sepal.Length Sepal.Width # 27.8 28.1 8.2 2.5 # Ranger filter (faster implementation) filt <- ranger_filter( y = y, x = x, nfilter = 3, num.trees = 500 ) ``` -------------------------------- ### Outer Cross-Validation with Random Forest Source: https://context7.com/myles-lewis/nestedcv/llms.txt Fits a Random Forest model using outer cross-validation. It requires the outcome variable (y), predictor variables (x), model type ('randomForest'), number of outer folds, and optionally the number of cores for parallel processing. ```r fit_rf <- outercv( y = y, x = x, model = "randomForest", n_outer_folds = 10, cv.cores = 2 ) summary(fit_rf) # Plot ROC curve plot(fit_rf$roc) ``` -------------------------------- ### Inner CV ROC Curve Extraction Source: https://context7.com/myles-lewis/nestedcv/llms.txt Explains how to extract and compare the ROC curve from inner cross-validation folds (`innercv_roc`) with the outer cross-validation ROC curve (`fit$roc`). This highlights the optimistic bias introduced by hyperparameter tuning during inner CV. ```r library(nestedcv) fit <- nestcv.glmnet(y, x, family = "binomial", cv.cores = 2) # Inner CV ROC (biased - hyperparameters optimized on this data) inner_roc <- innercv_roc(fit) # Outer CV ROC (unbiased) outer_roc <- fit$roc # Compare AUCs library(pROC) auc(outer_roc) # unbiased estimate auc(inner_roc) # optimistically biased # Visual comparison plot(outer_roc, col = "blue", main = "ROC Comparison") lines(inner_roc, col = "red") legend("bottomright", legend = c("Outer CV (unbiased)", "Inner CV (biased)"), col = c("blue", "red"), lty = 1) ``` -------------------------------- ### Outer Cross-Validation with Feature Filtering Source: https://context7.com/myles-lewis/nestedcv/llms.txt Performs outer cross-validation with a Random Forest model, incorporating feature filtering. It requires outcome (y), predictors (x), model type, a filtering function (e.g., lm_filter), filter options, and optionally parallel cores. ```r fit_filtered <- outercv( y = y_cont, x = x, model = "randomForest", filterFUN = lm_filter, filter_options = list(nfilter = 2, p_cutoff = NULL), cv.cores = 2 ) ``` -------------------------------- ### Cross-Validation of Alpha and Lambda for glmnet Source: https://context7.com/myles-lewis/nestedcv/llms.txt Performs k-fold cross-validation for glmnet, optimizing both the alpha mixing parameter and lambda. It requires predictor matrix (x), outcome vector (y), family type, a set of alpha values to test, and the number of folds. ```r library(nestedcv) # Prepare data data(iris) x <- as.matrix(iris[, 1:4]) y <- iris$Species # Cross-validate alpha and lambda cva_fit <- cva.glmnet( x = x, y = y, family = "multinomial", alphaSet = seq(0, 1, 0.1), # full range of alpha nfolds = 10 ) # View best alpha cva_fit$best_alpha # Access optimal CV mean error for each alpha cva_fit$alpha_cvm # Extract coefficients at optimal alpha/lambda coef(cva_fit, s = "lambda.min") # Make predictions preds <- predict(cva_fit, newx = x, s = "lambda.min", type = "class") # Plot alpha vs deviance with error bars plot(cva_fit, xaxis = "alpha") # Plot lambda vs deviance for all alphas plot(cva_fit, xaxis = "lambda") # Plot number of variables vs deviance plot(cva_fit, xaxis = "nvar") ``` -------------------------------- ### T-test Based Feature Selection Filter Source: https://context7.com/myles-lewis/nestedcv/llms.txt Implements a univariate t-test filter for binary classification, ranking variables by statistical significance. It can return indices, names, or full results including statistics and p-values. Options include specifying the number of features or a p-value cutoff. ```r library(nestedcv) # Binary classification data data(iris) x <- iris[iris$Species != "setosa", 1:4] y <- droplevels(iris$Species[iris$Species != "setosa"]) # Return indices of top 3 predictors by t-test p-value filt_idx <- ttest_filter(y, x, nfilter = 3) filt_idx # [1] 3 4 1 # Return predictor names instead filt_names <- ttest_filter(y, x, nfilter = 3, type = "names") filt_names # [1] "Petal.Length" "Petal.Width" "Sepal.Length" # Full results with p-values ttest_filter(y, x, type = "full") # stat pvalue # Sepal.Length -5.63 3.02e-07 # Sepal.Width -3.21 1.75e-03 # Petal.Length -27.6 2.45e-47 # Petal.Width -24.0 3.26e-42 # With p-value cutoff instead of fixed number filt <- ttest_filter(y, x, nfilter = NULL, p_cutoff = 0.001) # Remove collinear predictors filt <- ttest_filter(y, x, nfilter = 10, rsq_cutoff = 0.9) ``` -------------------------------- ### Nested CV with glmnet for Multinomial Family Source: https://github.com/myles-lewis/nestedcv/blob/main/README.md Performs nested cross-validation for a glmnet model with a multinomial family outcome. It tunes both lambda and alpha parameters using 10x10-fold CV on the iris dataset. Requires the nestedcv package and uses parallel processing for speed. ```r library(nestedcv) data(iris) y <- iris$Species x <- as.matrix(iris[, -5]) cores <- parallel::detectCores(logical = FALSE) # detect physical cores res <- nestcv.glmnet(y, x, family = "multinomial", cv.cores = cores) summary(res) ``` -------------------------------- ### Nested CV with caret Models in R Source: https://context7.com/myles-lewis/nestedcv/llms.txt Applies nested cross-validation to any model available through the caret package. This function allows users to leverage a wide array of machine learning algorithms within the nested CV framework, including options for embedded feature selection and custom tuning grids. It's useful for comprehensive model evaluation and comparison. ```r library(nestedcv) library(caret) # Prepare data data(iris) y <- iris$Species x <- iris[, 1:4] # Random forest with nested CV fit_rf <- nestcv.train( y = y, x = x, method = "rf", # random forest n_outer_folds = 5, n_inner_folds = 5, cv.cores = 2 ) summary(fit_rf) # glmnet via caret with custom tuning grid tg <- expand.grid( lambda = exp(seq(log(2e-3), log(1e0), length.out = 50)), alpha = seq(0.8, 1, 0.1) ) fit_glmnet <- nestcv.train( y = y, x = x, method = "glmnet", tuneGrid = tg, filterFUN = anova_filter, filter_options = list(nfilter = 3), cv.cores = 2 ) # SVM with radial basis kernel fit_svm <- nestcv.train( y = y, x = x, method = "svmRadial", tuneLength = 5, # 5 values for each tuning parameter cv.cores = 2 ) # Gradient boosting machine fit_gbm <- nestcv.train( y = y, x = x, method = "gbm", verbose = FALSE, cv.cores = 4 ) # Access best tuning parameters across folds fit_rf$bestTunes # Make predictions preds <- predict(fit_rf, newdata = x) # Plot inner CV tuning for outer fold 1 plot(fit_glmnet$outer_result[[1]]$fit, xTrans = log) ``` -------------------------------- ### Other Univariate Feature Selection Filters Source: https://context7.com/myles-lewis/nestedcv/llms.txt Provides various univariate filters for different data types and outcome variables, including Wilcoxon test for binary classification, ANOVA for multiclass, and Pearson/Spearman correlation for regression. The lm_filter adjusts for confounders. ```r library(nestedcv) # Wilcoxon (Mann-Whitney) test for binary classification filt <- wilcoxon_filter(y, x, nfilter = 3) # ANOVA filter for multiclass classification data(iris) y_multi <- iris$Species x_multi <- iris[, 1:4] filt <- anova_filter(y_multi, x_multi, nfilter = 3) # Pearson correlation filter for regression y_cont <- rnorm(150) x_cont <- matrix(rnorm(150 * 100), 150, 100) filt <- correl_filter(y_cont, x_cont, nfilter = 10, method = "pearson") # Spearman correlation filt <- correl_filter(y_cont, x_cont, nfilter = 10, method = "spearman") # Linear model filter with covariates # Useful when adjusting for confounders filt <- lm_filter( y = y_cont, x = x_cont, force_vars = c("V1", "V2"), # always include these nfilter = 20, p_cutoff = 0.05 ) ``` -------------------------------- ### Nested CV with Elastic Net (glmnet) in R Source: https://context7.com/myles-lewis/nestedcv/llms.txt Performs nested cross-validation using the glmnet package for elastic net regularization. It tunes both the alpha mixing parameter and lambda, and can optionally include embedded feature selection filters. Suitable for binary, multinomial classification, and regression tasks, especially when the number of predictors (P) is much larger than the sample size (n). ```r library(nestedcv) # Binary classification with P >> n set.seed(123) x <- matrix(rnorm(150 * 2000), 150, 2000) y <- factor(rbinom(150, 1, 0.5)) # Basic nested CV with glmnet fit <- nestcv.glmnet( y = y, x = x, family = "binomial", alphaSet = seq(0.7, 1, 0.1), # tune alpha from 0.7 to 1.0 n_outer_folds = 10, # 10-fold outer CV n_inner_folds = 10, # 10-fold inner CV cv.cores = 4 # parallel processing ) # View results summary(fit) # Output: AUC, Accuracy, Balanced accuracy with optimal alpha/lambda # Extract final model coefficients coef(fit) # Predict on new data preds <- predict(fit, newdata = x, type = "response") # With embedded feature filtering (t-test filter) fit_filtered <- nestcv.glmnet( y = y, x = x, family = "binomial", filterFUN = ttest_filter, filter_options = list(nfilter = 100, p_cutoff = NULL), alphaSet = seq(0.7, 1, 0.1), cv.cores = 4 ) # Multinomial classification (iris dataset) data(iris) y_multi <- iris$Species x_multi <- as.matrix(iris[, 1:4]) fit_multi <- nestcv.glmnet( y = y_multi, x = x_multi, family = "multinomial", alphaSet = seq(0.5, 1, 0.1), cv.cores = 2 ) summary(fit_multi) # Regression with continuous outcome y_reg <- rnorm(150) fit_reg <- nestcv.glmnet( y = y_reg, x = x, family = "gaussian", alphaSet = 1, # LASSO cv.cores = 4 ) fit_reg$summary # Returns RMSE, R-squared ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.