### R Setup and Benchmarking Function

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

This code block initializes the R environment by clearing variables, loading essential libraries such as 'fastverse' and 'collapse', setting package-specific options, and defining a helper function 'bmark' for performance benchmarking using the 'bench' package. It ensures all necessary dependencies are installed and configured for subsequent operations.

```r
rm(list = ls()); gc()
r_opts <- options(prompt = "R> ", continue = "+  ", width = 77, digits = 4, useFancyQuotes = FALSE, warn = 1)

# Loading libraries and installing if unavailable
if(!requireNamespace("fastverse", quietly = TRUE)) install.packages("fastverse")
options(fastverse.styling = FALSE)
library(fastverse) # loads data.table, collapse, magrittr and kit (not used)
# Package versions used in the article: ...
pkg <- c("bench", "dplyr", "tidyr", "matrixStats", "janitor", "nycflights23")
if(!all(avail <- is_installed(pkg))) install.packages(pkg[!avail])

# Reset collapse options (if set)
oldopts <- set_collapse(nthreads = 1L, remove = NULL, stable.algo = TRUE, sort = TRUE,
             digits = 2L, stub = TRUE, verbose = 1L, mask = NULL, na.rm = TRUE)

# For presentation: removing whitespace around labels
vlabels(GGDC10S) <- trimws(vlabels(GGDC10S))

# This is the benchmark function
bmark <- function(...) {
  bench::mark(..., min_time = 3, check = FALSE) |> janitor::clean_names() |
  fselect(expression, min, median, mem_alloc, n_itr, n_gc, total_time) |
  fmutate(expression = names(expression)) |> dapply(as.character) |> qDF()
}
```

--------------------------------

### Install collapse from GitHub

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Installs the latest development version of the collapse package directly from its GitHub repository. This method requires compilation and may be less stable than CRAN releases.

```r
remotes::install_github("SebKrantz/collapse")
```

--------------------------------

### Install collapse Previous Versions

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Installs a specific previous version of the collapse package from the CRAN Archive. This is useful for reproducibility or testing older functionalities. Compilation from source is typically required.

```r
install.packages("https://cran.r-project.org/src/contrib/Archive/collapse/collapse_2.0.19.tar.gz", 
                 repos = NULL, type = "source")
```

--------------------------------

### Install collapse from CRAN

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Installs the current stable version of the collapse package directly from the Comprehensive R Archive Network (CRAN). This is the standard method for obtaining the latest released version.

```r
install.packages("collapse")
```

--------------------------------

### R Package Installation and Loading

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Checks for the presence of required R packages ('fastverse', 'collapse') and installs them if they are not available or if a specific version of 'collapse' is needed. It then loads the 'fastverse' meta-package, which includes 'collapse'.

```r
# Loading libraries and installing if unavailable
if(!requireNamespace("fastverse", quietly = TRUE)) install.packages("fastverse")

if(!requireNamespace("collapse", quietly = TRUE) || packageVersion("collapse") < "2.1.2") {
  install.packages("collapse_2.1.2.tar.gz", type = "source", repos = NULL)
}

options(fastverse.styling = FALSE)
library(fastverse) # loads data.table, collapse, magrittr and kit (not used)
```

--------------------------------

### Setup for NYC Flights dataset benchmarks

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Initializes the environment for benchmarking operations on the NYC Flights dataset. It loads necessary packages and sets the number of threads for `data.table` operations.

```R
fastverse_extend(nycflights23, dplyr, data.table); setDTthreads(4)
list(flights, airports, airlines, planes, weather) |> sapply(nrow)
```

--------------------------------

### Setup for Join Benchmarking

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Prepares the environment for join operation benchmarking by cleaning up unused variables and running garbage collection. It also shows a preview of the join process and resulting dimensions.

```r
flights |> join(weather, on = c("origin", "time_hour")) |> 
  join(planes, on = "tailnum") |> join(airports, on = c(dest = "faa")) |> 
  join(airlines, on = "carrier") |> dim() |> 
  capture.output() |> substr(1, 76) |> cat(sep = "\n")
```

```r
rm(list = setdiff(ls(), c("bmark", "vars", "r_opts", "oldopts")))
gc()
```

--------------------------------

### join() Example with Verbosity

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Demonstrates the enhanced verbosity of the `join()` function, showing how it reports the join type and the proportion of matched keys between the left and right data frames.

```R
join(data.frame(id = c(1, 2, 2, 4)), data.frame(id = c(rep(1,4), 2:3)))
#> left join: x[id] 3/4 (75%) <1.5:1st> y[id] 2/6 (33.3%)
#>   id
#> 1  1
#> 2  2
#> 3  2
#> 4  4
join(data.frame(id = c(1, 2, 2, 4)), data.frame(id = c(rep(1,4), 2:3)), multiple = TRUE)
#> left join: x[id] 3/4 (75%) <1.5:2.5> y[id] 5/6 (83.3%)
#>   id
#> 1  1
#> 2  1
#> 3  1
#> 4  1
#> 5  2
#> 6  2
#> 7  4
```

--------------------------------

### group_by_vars() with magrittr Example

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Demonstrates a more complex data manipulation pipeline using `group_by_vars()` with the `magrittr` pipe. This example shows how to group, extract unique values, apply mean and sum aggregations with custom naming, and combine results.

```R
library(magrittr)
set_collapse(mask = "manip") # for fgroup_vars -> group_vars

data %>%
  group_by_vars(ind1) %>%
  {
    add_vars(
      group_vars(., "unique"),
      get_vars(., ind2) %>% fmean(keep.g = FALSE) %>% add_stub("mean_"),
      get_vars(., ind3) %>% fsum(keep.g = FALSE) %>% add_stub("sum_")
    ) 
  }
```

--------------------------------

### Install collapse Development Version (R-universe)

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Installs a stable development version of the collapse package from the R-universe repository. This often provides newer features or bug fixes before they are released on CRAN. Binaries are available for Windows and Mac.

```r
install.packages("collapse", repos = "https://fastverse.r-universe.dev")
```

--------------------------------

### collapse 2.0.11: Base Pipe Usage

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

In version 2.0.11, the package documentation examples were updated to predominantly use the base R pipe (`|>`) instead of the `magrittr` pipe (`%>%`). This change aims to reduce external dependencies in examples and promote the use of native R features.

```R
# Example using base pipe (|>)
result <- data |> 
  fmean() |> 
  fsum()

# Previously, might have used magrittr pipe (%>%)
# result <- data %>%
#   fmean() %>%
#   fsum()
```

--------------------------------

### R Setup: Clear Environment and Set Options

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Initializes the R session by clearing all existing objects from the workspace and running the garbage collector. It also sets global R options for prompt, continuation lines, output width, digit precision, and quote handling.

```r
rm(list = ls()); gc()
r_opts <- options(prompt = "R> ", continue = "+  ", width = 77, digits = 4, useFancyQuotes = FALSE, warn = 1)
```

--------------------------------

### fgrowth R Time Series Example

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Demonstrates the `fgrowth` function for time series analysis, specifically calculating growth rates. This example applies `fgrowth` to `airmiles` data, omits `NA` values, and rounds the results for clarity.

```R
fgrowth(airmiles, n = 10, power = 1/10) |> na.omit() |> round(1)
```

--------------------------------

### R Benchmarking Setup with collapse and janitor

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Defines a benchmarking function 'bmark' that uses the 'bench' package to measure performance. It integrates 'janitor::clean_names' for cleaner output column names and 'collapse::fselect', 'collapse::fmutate', 'collapse::dapply', and 'collapse::qDF' for efficient data manipulation and formatting of benchmark results.

```r
# This is the benchmark function
bmark <- function(...) {
  bench::mark(..., min_time = 3, check = FALSE) |> janitor::clean_names() |
  fselect(expression, min, median, mem_alloc, n_itr, n_gc, total_time) |
  fmutate(expression = names(expression)) |> dapply(as.character) |> qDF()
}
```

--------------------------------

### ts_example_2 R Growth Rate Calculation

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

This example demonstrates subsetting data and calculating growth rates using `fgrowth`. It first extracts a subset of the `exports` data and then applies `fgrowth` with different lag specifications (`-1:3`) and time variables (`t = y`), showing how to handle time series calculations with grouping.

```R
.c(y, v) %=% fsubset(exports, c == "c1" & s == "s7", -c, -s)
print(y)
```

```R
fgrowth(v, t = y) |> round(2)
```

```R
fgrowth(v, -1:3, t = y) |> head(4)
```

--------------------------------

### example_growth_continued R Grouped Growth Calculation

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

This example demonstrates calculating grouped growth rates using `fgroup_by` and `G`. It shows two equivalent methods for computing a grouped growth variable (`gv`) within the `exports` dataset, confirming that both approaches yield identical results.

```R
A <- exports |> fgroup_by(c, s) |> fmutate(gv = G(v, t = y)) |> fungroup()
head(B <- exports |> fmutate(gv = G(v, g = list(c, s), t = y)), 4)
```

```R
identical(A, B)
```

--------------------------------

### group_by_vars() and collapg() Example

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Illustrates the usage of the new `group_by_vars()` function in conjunction with `collapg()` for grouped data manipulation and aggregation. It shows how to group by a variable and then apply custom aggregation functions.

```R
data |> group_by_vars(ind1) |> collapg(custom = list(fmean = ind2, fsum = ind3))
```

--------------------------------

### New Function: flm for Linear Regression Fitting

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Introduces `flm`, a function for bare-bones (weighted) linear regression fitting. It offers multiple efficient methods, including base R options (`.lm.fit`, `solve`, `qr`, `chol`), and implementations from `RcppArmadillo` or `RcppEigen` if installed.

```R
flm(formula, data, weights = NULL, method = c("lm.fit", "solve", "qr", "chol", "fastLmCpp", "fastLmEigen"), ...)
# Fits linear models using various efficient backend methods.
```

--------------------------------

### Advanced Time Series Growth Calculations in R

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Provides examples of advanced time series growth calculations using `fgrowth` with different lag specifications and time variables. It also shows how to apply these calculations within groups using `G` and `tfm`.

```r
fgrowth(v, t = y) |> round(2)
```

```r
fgrowth(v, -1:3, t = y) |> head(4)
```

```r
G(exports, -1:2, by = v ~ c + s, t = ~ y) |> head(3)
```

```r
tfm(exports, fgrowth(list(v = v), -1:2, g = list(c, s), t = y)) |> head(3)
```

--------------------------------

### Data Generation and Transformation in R

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Illustrates generating a large dataset using `expand.grid`, `fmutate` for adding random values, `colorder` for rearranging columns, and `fsubset` for random row removal. This sets up data for further analysis.

```r
set.seed(101)
exports <- expand.grid(y = 2001:2010, c = paste0("c", 1:10),
                       s = paste0("s", 1:10)) |> fmutate(v = abs(rnorm(1e3))) |>
  colorder(c, s) |> fsubset(-sample.int(1e3, 500))
```

--------------------------------

### R: Data Preparation and Summary with namlab

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Prepares a dataset by creating a new 'Label' column and subsetting based on finite values, then uses the 'namlab' function to display variable names, classes, counts, and distinct values. This is useful for data exploration.

```r
data <- GGDC10S |>
  fmutate(Label = ifelse(Variable == "VA", "Value Added", "Employment")) |>
  fsubset(is.finite(AGR), Country, Variable, Label, Year, AGR:MAN)
namlab(data, N = TRUE, Ndistinct = TRUE, class = TRUE)
```

--------------------------------

### Create and Subset Indexed Data in R

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Shows how to create an indexed data frame using `findex_by` and then subset it using `G` for grouped operations. It also demonstrates how to reset or view the index structure.

```r
exportsi <- exports |> findex_by(c, s, y)
exportsi |> G(0:1) |> head(5)
```

```r
exportsi |> findex() |> print(2)
```

--------------------------------

### R: `replace_NA` with `cols` Argument

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

The `replace_NA` function now accepts a `cols` argument, allowing users to specify which columns to apply NA replacement to, for example, by column type.

```R
library(collapse)

# Replace NAs only in numeric columns
data %>% replace_NA(cols = is.numeric)

# Replace NAs in specific columns
data %>% replace_NA(cols = c("col1", "col3"))

# Note: For large numeric data, data.table::setnafill is more efficient.

```

--------------------------------

### Inspect Data Dimensions and Distinct Values

Source: https://github.com/sebkrantz/collapse/blob/master/misc/arrow benchmark/arrow_benchmark.md

Uses fastverse functions fdim to get the dimensions (rows, columns) of a data.table and fndistinct to count the number of distinct values for each column.

```r
fdim(taxi)
#> [1] 10000000       17
fndistinct(taxi)
#>           vendor_id     pickup_datetime    dropoff_datetime     passenger_count
#>                   3             7824766             7823041                  10
#>       trip_distance           rate_code  store_and_fwd_flag        payment_type
#>                6502                   7                   2                   5
#>         fare_amount               extra             mta_tax          tip_amount
#>                8286                  61                  24                5919
#>        tolls_amount       imp_surcharge        total_amount  pickup_location_id
#>                2463                  12               20933                 262
#> dropoff_location_id
#>                 263

```

--------------------------------

### fsum and fprod Weight Argument

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

The `fsum` (sum) and `fprod` (product) functions have been updated to include a `w` argument, allowing for weighted summation and product calculations.

```APIDOC
fsum(x, w = NULL)
fprod(x, w = NULL)

Parameters:
  x: The numeric vector to sum or multiply.
  w: Optional weights for the elements in x. Defaults to NULL (unweighted).

Description:
  Adds support for weighted calculations to the `fsum` and `fprod` functions. This allows for more accurate statistical computations when observations have different levels of importance.

Example Usage:
  # Weighted sum
  fsum(c(1, 2, 3), w = c(0.5, 1, 2))

  # Weighted product
  fprod(c(2, 3, 4), w = c(0.5, 1, 2))
```

--------------------------------

### Fast Data Manipulation: Pivot and Join

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Demonstrates `collapse`'s data manipulation capabilities, including pivoting data wider using the `pivot` function and joining datasets. Shows how to reshape data and merge information from different sources.

```R
library(collapse)

# Pivot Wider: Only SUM (total)
SUM <- GGDC10S |> pivot(c("Country", "Year"), "SUM", "Variable", how = "wider")
head(SUM)

# Joining with data from wlddev
wlddev |
    join(SUM, on = c("iso3c" = "Country", "year" = "Year"), how = "inner")
```

--------------------------------

### Logical Operators with Missing Doubles

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Updates `whichv()` and the operators `%==%` and `%!=%` to correctly handle missing double values. For example, `c(NA_real_, 1) %==% c(NA_real_, 1)` now correctly yields `c(1, 2)` instead of `2`.

```R
whichv(x, y)
x %==% y
x %!=% y
```

--------------------------------

### Print Session Information

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Displays detailed session information for the current R environment, including R version, platform, locale, loaded packages, and their versions. This is crucial for reproducibility.

```R
###################################################
### Print Session Information
###################################################

sessionInfo()
```

--------------------------------

### Arrow Unsupported Operation (cumsum)

Source: https://github.com/sebkrantz/collapse/blob/master/misc/arrow benchmark/arrow_benchmark.md

Demonstrates an operation (`cumsum`) not supported by the Arrow backend in R. When encountered, Arrow pulls the data into R for computation, which can impact performance. This example shows the warning message and the resulting data structure.

```R
taxi_arrow |
  group_by(vendor_id, pickup_location_id, dropoff_location_id, passenger_count) |
  mutate(total_amount = cumsum(total_amount)) |> collect()

```

--------------------------------

### R collapse Package Topics and Help

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

This snippet displays the main topics covered by the 'collapse' R package and opens the package's main documentation page. It is useful for understanding the scope and features of the library, providing an overview of its capabilities.

```r
.COLLAPSE_TOPICS
help("collapse-documentation")
```

--------------------------------

### Evaluating Arbitrary Expressions in fsummarise and fmutate

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Details the new capability in `fsummarise()` and `fmutate()` to evaluate arbitrary expressions that result in lists or data frames without requiring `across()`. It provides examples using `mctl()` and demonstrates efficient column selection with `.cols`.

```r
# fsummarise and fmutate gained an ability to evaluate arbitrary expressions that result in lists / data frames without the need to use across().
# Example: mtcars |> fgroup_by(cyl) |> fsummarise(mctl(lmtest::coeftest(lm(mpg ~ wt + carb)), names = TRUE))
# Example with column selection: mtcars |> fgroup_by(cyl, vs, am) |> fsummarise(mctl(cor(.data), names = TRUE), .cols = .c(mpg, wt, carb))
```

--------------------------------

### R Joins: Inner Join with Duplicates

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Performs an inner join between 'teacher' and 'course' data frames on the 'id' column. This example handles multiple matches and manages duplicate column names by renaming them with a suffix to avoid conflicts.

```R
course$names <- teacher$names[course$id]
join(teacher, course, on = "id", how = "inner", multiple = TRUE)
```

--------------------------------

### Basic Data Selection and Unique Count in R

Source: https://github.com/sebkrantz/collapse/blob/master/misc/arrow benchmark/arrow_benchmark.md

Demonstrates selecting specific columns from a data frame and counting unique combinations using R. This snippet shows a common data preprocessing step.

```R
taxi |> fselect(vendor_id, pickup_location_id, passenger_count) |> fnunique()
#> [1] 3498
```

--------------------------------

### Benchmark match, chmatch, and fmatch for character vectors

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Compares the speed of finding elements in character vectors using base R's `match`, `data.table`'s `chmatch`, and `collapse`'s `fmatch`. This benchmark demonstrates `collapse`'s advantage in character matching.

```R
bmark(base = match(g_char, char), data.table = chmatch(g_char, char), collapse = fmatch(g_char, char))
```

--------------------------------

### gtrans R Data Transformation

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Illustrates data transformation using `add_vars` and `fwithin` from the `collapse` package. This example applies a within-group mean calculation and adds a stub prefix to the transformed variables, demonstrating flexible data manipulation.

```R
add_vars(wlddev) <- get_vars(wlddev, c("PCGDP", "LIFEEX")) |>
  fwithin(g, mean = "overall.mean") |> add_stub("center_")
```

--------------------------------

### R: Internal grouping for Fast Statistical Functions

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Starting from version 1.7.1, Fast Statistical Functions within 'collapse' now internally use 'group' for data grouping when both 'g' and 'TRA' arguments are provided. This optimization yields efficiency gains, particularly on unsorted data.

```R
fast_stat_function(..., g, TRA)
```

--------------------------------

### Simple Aggregation with collap

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Demonstrates basic data aggregation using the collap function, grouping by 'Species' and calculating the mean of 'Sepal.Length' and 'Sepal.Width'.

```R
collap(iris, Sepal.Length + Sepal.Width ~ Species, fmean)
```

--------------------------------

### Reduced Dependency on `lfe` Package

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

The dependency on the `lfe` package is now optional. Higher-dimensional centering in `fhdwithin` and `fhdbetween` requires `lfe`, but single-factor linear prediction and centering are still available without it. This reduces the package's installation burden and broadens compatibility down to R version 2.10.

```R
# fhdwithin(..., hdfc = TRUE)
# fhdbetween(..., hdfc = TRUE)
# Require 'lfe' package for higher-dimensional centering.
# Base R and Rcpp are the only mandatory dependencies.
```

--------------------------------

### Ad-hoc Transformations in Grouped_df Methods

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Highlights the support for ad-hoc transformations in the 't' and 'w' arguments for 'grouped_df' methods, including formula input. Provides examples using `gby`, `flag`, `L`, `qsu`, `collapg`, and `nv` with transformations like `qG(date)` and `log(POP)`.

```R
### Ad-hoc Transformations in Grouped_df Methods
* 't' and 'w' arguments in 'grouped_df' methods support ad-hoc transformations.
* Supports formula input for transformations.
* Examples:
  - `wlddev %>% gby(iso3c) %>% flag(t = qG(date))`
  - `L(wlddev, 1, ~ iso3c, ~qG(date))`
  - `qsu(wlddev, w = ~ log(POP))`
  - `wlddev %>% gby(iso3c) %>% collapg(w = log(POP))`
  - `wlddev %>% gby(iso3c) %>% nv() %>% fmean(w = log(POP))`
```

--------------------------------

### Piped Programming with collapse and magrittr

Source: https://github.com/sebkrantz/collapse/blob/master/README.md

Shows how to leverage `collapse` functions within a `magrittr` pipe for efficient data manipulation. Demonstrates grouped distinct counts, weighted group medians, subsetting, and calculating group variance with weights.

```R
library(collapse)
library(magrittr)                            # Pipe operators
iris %>% fgroup_by(Species) %>% fndistinct   # Grouped distinct value counts
iris %>% fgroup_by(Species) %>% fmedian(w)   # Weighted group medians 
iris %>% add_vars(w) %>%                     # Adding weight vector to dataset
  fsubset(Sepal.Length < fmean(Sepal.Length), Species, Sepal.Width:w) %>% # Fast selecting and subsetting
  fgroup_by(Species) %>%                     # Grouping (efficiently creates a grouped tibble)
  fvar(w) %>%                                # Frequency-weighted group-variance, default (keep.w = TRUE)  
  roworder(sum.w)                            # also saves group weights in a column called 'sum.w'
```

--------------------------------

### Get Variable Labels from Data Frame

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Retrieves specific variable labels from a data frame `dr`. It accesses elements from index 3 to 6 of the output of `vlabels()`. This function is likely used for inspecting metadata or column names.

```R
vlabels(dr)[3:6]
```

--------------------------------

### Unordered Grouping Algorithm

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Introduces a new low-level unordered grouping algorithm implemented in C, inspired by the 'kit' package. It optimizes R's data structures and is used internally by functions like GRP, fgroup_by, funique, qF, and fwithin. Performance is noted as promising, often superior to radixorder.

```R
group(data, sort = FALSE)
GRP(data, sort = FALSE)
fgroup_by(data, sort = FALSE)
funique(data, sort = FALSE)
qF(data, sort = FALSE)
fwithin(data, g = NULL)
```

--------------------------------

### Load and Extend fastverse Libraries

Source: https://github.com/sebkrantz/collapse/blob/master/misc/arrow benchmark/arrow_benchmark.md

Loads the fastverse meta-package and extends it with additional libraries like dplyr and arrow. It also shows common conflicts resolved by fastverse.

```r
# System: M1 MAC (2020), 16GB, CRAN Binaries
library(fastverse)
#> -- Attaching packages --------------------------------------- fastverse 0.2.4 --
#> collapse       1.8.7      testthat       3.1.4
#> magrittr       2.0.3      microbenchmark 1.4.9
#> data.table     1.14.2     kit            0.0.12
#> -- Conflicts ------------------------------------------ fastverse_conflicts() --
#> testthat::equals()       masks magrittr::equals()
#> testthat::is_less_than() masks magrittr::is_less_than()
#> testthat::not()          masks magrittr::not()
fastverse_extend(dplyr, arrow, microbenchmark)
#> -- Attaching extension packages ----------------------------- fastverse 0.2.4 --
#> dplyr 1.0.9     arrow 8.0.0
#> -- Conflicts ------------------------------------------ fastverse_conflicts() --
#> dplyr::between()   masks data.table::between()
#> dplyr::filter()    masks stats::filter()
#> dplyr::first()     masks data.table::first()
#> dplyr::intersect() masks base::intersect()
#> arrow::is_in()     masks magrittr::is_in()
#> dplyr::lag()       masks stats::lag()
#> dplyr::last()      masks data.table::last()
#> arrow::matches()   masks dplyr::matches(), testthat::matches()
#> dplyr::setdiff()   masks base::setdiff()
#> dplyr::setequal()  masks base::setequal()
#> arrow::union()     masks base::union()

```

--------------------------------

### Quantify and Summarize Data

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Applies the `qsu` function to `LIFEEXi`, which is likely a function for quantifying and summarizing data, possibly calculating statistics like mean, variance, or quantiles. This is used to get a statistical overview of the indexed life expectancy data.

```R
qsu(LIFEEXi)
```

--------------------------------

### R collapse: Fast Scaling and Standard Deviation

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Demonstrates 'fscale' for fast standardization (centering and scaling) of a numeric vector within groups, and 'fsd' for calculating the standard deviation within groups. The example verifies that scaling results in a standard deviation of 1 for each group.

```r
fscale(iris$Sepal.Length, g = iris$Species) |> fsd(g = iris$Species)
```

--------------------------------

### indexing R Data Indexing and Subsetting

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Demonstrates data indexing and subsetting capabilities in `collapse`. It uses `findex_by` to create an indexed dataset and then applies `G` with different lag specifications and `findex` to extract specific subsets and view index information.

```R
exportsi <- exports |> findex_by(c, s, y)
exportsi |> G(0:1) |> head(5)
```

```R
exportsi |> findex() |> print(2)
```

--------------------------------

### R Joins: Semi and Anti Joins

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Demonstrates performing semi and anti joins between 'teacher' and 'course' data frames using the 'join' function. It iterates through 'semi' and 'anti' join types to show their respective outputs.

```R
for (h in c("semi", "anti")) join(teacher, course, how = h) |> print()
```

--------------------------------

### Row- and Column-wise Arithmetic Operators

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Introduces a set of 10 operators (`%rr%`, `%r+%`, `%r-%`, `%r*%`, `%r/%`, `%cr%`, `%c+%`, `%c-%`, `%c*%`, `%c/%`) for efficient row- and column-wise arithmetic operations between vectors and matrices/data frames/lists. For example, `X %r*% v` multiplies each row of `X` by `v`.

```R
## New arithmetic operators for matrix/vector operations:
# Operators: %rr%, %r+%, %r-%, %r*%, %r/%, %cr%, %c+%, %c-%, %c*%, %c/%
# Example: X %r*% v multiplies each row of matrix X by vector v.
# Facilitates intuitive and convenient matrix-style operations.
```

--------------------------------

### R: Global Settings with set_collapse() and get_collapse()

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Provides functions to globally set and retrieve default options for the collapse package, such as `nthreads` and `na.rm`. These settings can be configured via `.Rprofile` or `.fastverse` files for consistent behavior across sessions. The package uses an internal environment `.op` for efficient access to these defaults.

```R
# Set global defaults
set_collapse(nthreads = 4, na.rm = FALSE)

# Retrieve current settings
get_collapse("nthreads")
.op[["nthreads"]]

# Note: Certain internal functions like .quantile and .range bypass these settings for maximum performance.
```

--------------------------------

### alloc Object Replication

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Describes the expanded capabilities of the alloc function, which can now replicate any object type. It handles NULL, non-atomic, or atomic inputs with length > 1 by returning a list of replicated objects.

```R
alloc(NULL, 10) # Returns a length 10 list of NULL objects
alloc(mtcars, 10) # Returns a list of 10 mtcars datasets (not deep-copied)
```

--------------------------------

### R Grouped Aggregation with BY

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Showcases the 'BY' function for performing custom aggregations on data. This example applies the 'mad' (median absolute deviation) function to multiple numeric variables within groups defined by 'iris$Species'. The output is a data frame with aggregated statistics.

```r
BY(num_vars(iris), g = iris$Species, FUN = mad) |> head(1)
```

--------------------------------

### example_growth R Growth Rate Calculation with Grouping

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Illustrates calculating growth rates using the `G` function, which is a wrapper for `fgrowth` with explicit grouping. It shows how to apply growth calculations across different groups (`by = v ~ c + s`) and time variables (`t = ~ y`), demonstrating flexible panel data analysis.

```R
G(exports, -1:2, by = v ~ c + s, t = ~ y) |> head(3)
```

```R
tfm(exports, fgrowth(list(v = v), -1:2, g = list(c, s), t = y)) |> head(3)
```

--------------------------------

### New Function: qTBL for Quick Tibble Conversion

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Presents the `qTBL` function, a utility for quickly converting various R objects into tibbles. This simplifies data wrangling and ensures consistent data frame structures.

```R
qTBL(x, ...)
# Converts R objects to tibbles efficiently.
```

--------------------------------

### grouped_df Methods for flag, fdiff, fgrowth with Multiple Time Variables

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Enhances the `grouped_df` methods for `flag`, `fdiff`, and `fgrowth` to support multiple time variables for panel identification. This allows for more complex panel data analysis, as shown in the example `data %>% fgroup_by(region, person_id) %>% flag(1:2, list(month, day))`.

```R
data %>%
  fgroup_by(region, person_id) %>%
  flag(1:2, list(month, day))
# Supports multiple time variables for panel data analysis.
```

--------------------------------

### R: `ftransform`/`fcompute` Efficiency and `keep` Argument

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Updates `ftransform` and `fcompute` to use `alloc` instead of `base::rep` for scalar replication, offering slight efficiency gains. `fcompute` now includes a `keep` argument to preserve existing columns when computing new ones.

```R
library(collapse)

# Using alloc for scalar replication (more efficient)
# Example: Creating a new column with a constant value
data %>% ftransform(new_col = 1)

# Using the 'keep' argument in fcompute
# Preserves 'id' and 'value' columns while computing 'new_feature'
data %>% 
  fcompute(new_feature = var1 * var2, 
           keep = c("id", "value"))

```

--------------------------------

### R collapse: Fast Mean with TRA = "fill"

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/jss5278/jss5278.md

Illustrates the 'fmean' function with the 'TRA = "fill"' argument. This option fills missing group means with the overall mean, useful for imputation or consistent group-wise calculations. The example shows the first and last few results for clarity.

```r
fmean(iris$Sepal.Length, g = iris$Species, TRA = "fill")[c(1:5, 51:55)]
```

--------------------------------

### Add formula methods for flm and fFtest

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Provides formula interface methods for `flm` and `fFtest`, in addition to the existing programming interface. This allows users to specify models using R's formula syntax.

```R
flm(mpg ~ hp + carb, mtcars, weights = wt)
fFtest(mpg ~ hp + carb | vs + am, mtcars, weights = wt)
```

--------------------------------

### R Fast Scaling and Standard Deviation (fscale, fsd)

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Demonstrates the 'fscale' function for standardizing data (centering and scaling) and 'fsd' for calculating the standard deviation, both with grouping capabilities. This example scales the 'Sepal.Length' column by 'iris$Species' and then calculates the standard deviation of the scaled values within each species group, expecting a result of 1.

```r
fscale(iris$Sepal.Length, g = iris$Species) |> fsd(g = iris$Species)
```

--------------------------------

### Grouped Growth Rate Calculation and Comparison in R

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Demonstrates calculating a grouped growth rate (`gv`) using `fgroup_by` and `fmutate` with `G`, and then compares it to a direct calculation using `fmutate` with `G` specifying groups. It verifies the equivalence of both methods.

```r
A <- exports |> fgroup_by(c, s) |> fmutate(gv = G(v, t = y)) |> fungroup()
head(B <- exports |> fmutate(gv = G(v, g = list(c, s), t = y)), 4)
```

```r
identical(A, B)
```

--------------------------------

### R Fast Mean with TRA="fill" (fmean)

Source: https://github.com/sebkrantz/collapse/blob/master/misc/JSS article/final/article_replication.md

Illustrates the 'fmean' function with the 'TRA = "fill"' argument, which fills missing values within groups using the group's mean. This example shows how to apply this functionality to a column grouped by another factor, demonstrating a common data imputation or transformation technique.

```r
fmean(iris$Sepal.Length, g = iris$Species, TRA = "fill")[c(1:5, 51:55)]
```

--------------------------------

### R: Fast Data Subsetting with fsubset and ss

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Adds `fsubset` as a much faster alternative to `base::subset` for subsetting vectors, matrices, and data frames. Also introduces `ss` as a faster alternative to `[.data.frame`.

```R
fsubset(x, subset, select, ...)
# Faster version of base::subset.

ss(data, row_indices, col_indices)
# Faster alternative to data[row_indices, col_indices, drop = FALSE].
```

--------------------------------

### Bug Fix: across with Multiple Functions

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

A bug in the `across` function when provided with multiple functions in a list has been resolved. The issue occurred when two `function(x)` statements were passed, for example, in `fsummarise(acr(mpg, list(ssdd = function(x) sd(x), mu = function(x) mean(x))))`. This fix ensures correct processing of multiple anonymous functions within `across`. Reported in issue #233.

```R
## Previously problematic syntax:
# mtcars |> fsummarise(acr(mpg, list(ssdd = function(x) sd(x), mu = function(x) mean(x))))

## After fix:
# The above syntax now works as expected.
```

--------------------------------

### Add as.data.frame.qsu method

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Implements an `as.data.frame` method for `qsu` objects, enabling efficient conversion of default array outputs from `qsu()` into tidy data frames.

```R
as.data.frame.qsu(qsu_output)
```

--------------------------------

### vlabels/setLabels: C Rewrite for Speed

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

vlabels and setLabels have been rewritten in C, resulting in approximately a 20x speed improvement. These functions now operate by reference.

```R
vlabels(x) <- "new label"
setLabels(x, "new label")
# Operations are now by reference.
```

--------------------------------

### R: Direct Calling of `settransform`/`settransformv`

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

The `settransform` and `settransformv` functions can now be called directly using the `collapse::` namespace without requiring the package to be explicitly loaded. This resolves previous errors when the package was not attached.

```R
library(collapse) # Or without loading if calling directly

# Previously required library(collapse) to be loaded
# data <- ftransform(data, new_col = var1 * 2)

# Now callable directly via namespace
data <- collapse::ftransform(data, new_col = var1 * 2)

# Similarly for settransformv
# data <- collapse::settransformv(data, c("var1", "var2"), c("new_var1", "new_var2"))

```

--------------------------------

### Project Configuration: .onLoad and collapse_mask

Source: https://github.com/sebkrantz/collapse/blob/master/NEWS.md

Explains the new `.onLoad` functionality that checks for a `.fastverse` configuration file to apply `_opt_collapse_mask`. This ensures that function masking is active when the package loads, enhancing security and consistency for projects using fastverse masking.

```R
### Project Configuration and Masking
* `.onLoad` checks for `.fastverse` config file with `_opt_collapse_mask` setting.
* Ensures function masking is applied before package load.
* Enhances security for projects using fastverse function masking.
* See `help("collapse-options")` and fastverse vignette for details.
```