### Set Up Local Pre-Commit Hooks Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md These commands install the pre-commit framework and then activate the configured pre-commit hooks for the current repository. Pre-commit hooks automate code quality checks before committing changes, helping to catch issues early in the development cycle. ```bash pip install pre-commit pre-commit install ``` -------------------------------- ### Providing Function Examples with @examples (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Illustrates how to include runnable examples for an R function using the `@examples` tag in `roxygen2`. It shows basic usage and how to wrap slower or conditional examples with `\donttest{}`. ```R #' @examples #' # Basic usage #' x <- c(1, 2, 3, NA, 5) #' compute_stats(x, na.rm = TRUE) #' #' # With matrix input #' m <- matrix(1:12, nrow = 3) #' compute_stats(m) #' #' \donttest{ #' # Slower example (>5 seconds or requires special conditions) #' if (requireNamespace("largedata", quietly = TRUE)) { #' # Simulating data generation for example #' # big_data <- largedata::generate_data(1e6) #' # compute_stats(big_data, method = "robust") #' } #' } ``` -------------------------------- ### Install R Development Dependencies for fmristore Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md This command installs all necessary R package dependencies required for developing the fmristore package. It uses the devtools package to ensure all development-specific packages are available, streamlining the setup process for contributors. ```R devtools::install_dev_deps() ``` -------------------------------- ### Install fmristore R package from GitHub Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs the development version of the `fmristore` R package directly from GitHub using `devtools`. This command requires the `devtools` package to be installed first. ```R devtools::install_github("bbuchsbaum/fmristore") ``` -------------------------------- ### Install and set up pre-commit hooks for R development Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs the `pre-commit` framework (requires Python) and sets up Git hooks to automate code style checks and formatting with `styler` and `lintr` before commits. Includes an optional command to run hooks against all files in the repository. ```bash pip install pre-commit pre-commit install pre-commit run --all-files ``` -------------------------------- ### Using usethis for R Package Setup and Licensing Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Demonstrates common `usethis` package functions to streamline the setup of R packages, including package-level documentation, data inclusion, vignette creation, licensing, and testing infrastructure. ```R usethis::use_package_doc() usethis::use_data() usethis::use_vignette("my-vignette-title") usethis::use_mit_license() usethis::use_gpl3_license() usethis::use_testthat() ``` -------------------------------- ### Install HDF5 system libraries on Ubuntu/Debian Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs the HDF5 development libraries required by the `fmristore` package on Ubuntu or Debian-based Linux systems using the `apt-get` package manager. ```bash sudo apt-get install libhdf5-dev ``` -------------------------------- ### R Package Examples: Safe File System Operations with tempdir() Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md This snippet provides a template for writing R package examples that interact with the file system safely. It demonstrates using `tempdir()` to create temporary directories and files, ensuring examples are CRAN-compliant by avoiding permanent file system modifications and including cleanup. ```R #' @examples #' \dontshow{ #' # CRAN-safe temporary directory for example output #' old_wd <- setwd(tempdir()) #' old_dir_to_create <- "my_temp_example_dir" #' if (!dir.exists(old_dir_to_create)) dir.create(old_dir_to_create) #' } #' #' # Example that writes a file #' my_data_frame <- data.frame(a = 1:3, b = letters[1:3]) #' # write.csv(my_data_frame, file.path(old_dir_to_create, "temp_file.csv")) #' # message("File written to: ", file.path(tempdir(), old_dir_to_create, "temp_file.csv")) #' #' \dontshow{ #' # Cleanup: remove created files/directories and restore working directory #' # unlink(file.path(old_dir_to_create, "temp_file.csv")) #' # unlink(old_dir_to_create, recursive = TRUE) #' setwd(old_wd) #' } ``` -------------------------------- ### Install HDF5 system libraries on macOS with Homebrew Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs the HDF5 development libraries required by the `fmristore` package on macOS using the Homebrew package manager. ```bash brew install hdf5 ``` -------------------------------- ### Install HDF5 system libraries on macOS with MacPorts Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs the HDF5 development libraries required by the `fmristore` package on macOS using the MacPorts package manager. ```bash sudo port install hdf5 ``` -------------------------------- ### Load fmristore R package for quick start Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Loads the `fmristore` R package into the current R session, making its functions and data structures available for use in neuroimaging data analysis workflows. ```R library(fmristore) # Load neuroimaging data # Example workflow here... ``` -------------------------------- ### Create Compressed HDF5 Dataset Example with hdf5r Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt This comprehensive R example demonstrates the process of creating an HDF5 file, defining a dataset creation property list with chunking and Gzip compression (level 9), creating a new dataset with random data, and finally closing the HDF5 file. ```R library(hdf5r) # Create HDF5 file file <- H5File$new("example.h5", mode = "w") # Create dataset creation property list with compression dcpl <- H5P_DATASET_CREATE$new() dcpl$set_chunk(c(100, 100))$set_deflate(9) # Create dataset with compression data_matrix <- matrix(runif(10000), nrow = 100, ncol = 100) file$create_dataset( name = "compressed_dataset", data = data_matrix, dtype = h5types$H5T_IEEE_F32LE, dataset_create_pl = dcpl ) # Close the file file$close_all() ``` -------------------------------- ### R Documentation Conditional Examples Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Illustrates how to write R package examples that conditionally execute based on dependencies or environment, using `\donttest{}` to prevent full execution during CRAN checks and `if` statements for conditional logic. ```R #' @examples #' # Basic example (always runs) #' basic_function_call(1:10) #' #' \donttest{ #' # Example depending on a suggested package #' if (requireNamespace("ggplot2", quietly = TRUE)) { #' # data_for_plot <- ... #' # ggplot2::ggplot(data_for_plot) + ggplot2::geom_point() #' } #' #' # Example requiring internet (and ideally interactive session) #' if (interactive() && curl::has_internet()) { #' # downloaded_content <- download_function_from_web("https://example.com/data.txt") #' } #' } ``` -------------------------------- ### Run tests and check coverage for fmristore R package Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Installs development dependencies, executes the full test suite, and generates a code coverage report for the `fmristore` R package using `devtools` and `covr`. ```R devtools::install_dev_deps() devtools::test() covr::package_coverage() ``` -------------------------------- ### Browse Vignettes for an Installed R Package Source: https://github.com/bbuchsbaum/fmristore/blob/main/vignettes/README.md This R command opens a web browser displaying a list of all vignettes associated with a specified installed R package. Users can then click on individual vignettes to view their content, providing easy access to package documentation. ```R browseVignettes("fmristore") ``` -------------------------------- ### APIDOC: Transform Implementation Guide Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Provides guidelines for implementing S3 methods for transforms, including retrieving schema defaults, understanding `write_mode` fallback behavior, and implementing subsetting logic. ```APIDOC Transform Implementation Guide: - Implement S3 methods. - Use `lna:::default_params(type)` to retrieve schema defaults. - Be aware `write_mode = "stream"` falls back to "eager" in v1.4. - Implement subsetting logic. ``` -------------------------------- ### Run All R Package Tests Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md This command executes the entire test suite for the R package. It's crucial to run this before committing to ensure no regressions are introduced and all functionalities work as expected, maintaining the package's stability. ```R devtools::test() ``` -------------------------------- ### Documenting an S3 Generic Function in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Provides an example of documenting an S3 generic function summarize_data in R. It includes @param, @return, and @examples tags, demonstrating how to define a generic function that dispatches to specific methods based on the input object's class. ```R #' Summarize Data Objects #' #' Generic function for creating summaries of various data types. #' #' @param x Object to summarize. #' @param ... Additional arguments passed to methods. #' @return A summary object (class depends on input type). #' @export #' @examples #' # Example for a numeric vector #' summarize_data(c(1, 2, 3, NA, 5), na.rm = TRUE) #' #' # Example for a data frame (will dispatch to data.frame method) #' df <- data.frame(a = 1:3, b = letters[1:3]) #' summarize_data(df) summarize_data <- function(x, ...) { UseMethod("summarize_data") } ``` -------------------------------- ### Build and Check R Package Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Commands to build, check, and install the R package locally, ensuring CRAN compliance. Includes both shell commands and R `devtools` functions. ```bash R CMD build . R CMD check --as-cran fmristore_*.tar.gz R CMD INSTALL . ``` ```R devtools::install() devtools::check() devtools::document() ``` -------------------------------- ### Enforce Tidyverse R Code Style and Linting Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md These R commands help maintain code quality by enforcing the tidyverse style guide. `styler::style_pkg()` automatically formats the package code, while `lintr::lint_package()` identifies any remaining style violations or potential issues, ensuring consistent and readable code. ```R styler::style_pkg() # Auto-fix formatting lintr::lint_package() # Check for violations ``` -------------------------------- ### Documenting Function Return Value with @return (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Demonstrates the use of the `@return` tag in `roxygen2` to describe the output of an R function. It includes an example of documenting a complex list structure using `\item{}`. ```R #' @return A named list with elements: #' \item{mean}{Arithmetic mean.} #' \item{median}{Median value.} #' \item{sd}{Standard deviation.} #' Returns NULL if input is empty after NA removal. ``` -------------------------------- ### R: Get or Set Global lna Options Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md This R function allows users to get or set global `lna` options. Options can be set using named arguments (e.g., `write.compression_level=4`) or retrieved by providing their character names. If no arguments are provided, it returns all current options. ```R lna_options <- function(...) { # Uses internal package environment } ``` -------------------------------- ### Debug R Package Test Failures Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md These R commands provide tools for debugging test failures. You can run specific test files, filter tests by name for focused debugging, or check code coverage for individual files to identify untested areas, aiding in efficient problem resolution. ```R # Run specific test file devtools::test_file("tests/testthat/test-file.R") # Run with debugging devtools::test(filter = "test_name") # Check coverage for specific file covr::file_coverage("R/file.R", "tests/testthat/test-file.R") ``` -------------------------------- ### Troubleshoot and Fix R Code Style Issues Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md These R commands assist in diagnosing and resolving code style violations. `styler::style_pkg(dry = "on")` shows potential changes without applying them, `styler::style_pkg()` applies automatic fixes, and `lintr::lint_package()` identifies any remaining manual fixes needed to ensure style compliance. ```R # See what styler would change styler::style_pkg(dry = "on") # Apply fixes styler::style_pkg() # Check remaining violations lintr::lint_package() ``` -------------------------------- ### Check R Package Test Coverage Source: https://github.com/bbuchsbaum/fmristore/blob/main/DEVELOPMENT.md This command calculates the test coverage for the entire R package. It's used to ensure that a sufficient percentage of the code is covered by automated tests, contributing to overall code quality and reducing the risk of undetected bugs. ```R covr::package_coverage() ``` -------------------------------- ### Documenting an R Dataset Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Illustrates how to document an R dataset (survey_data) for inclusion in a package. It details the use of @format to describe the data structure, @describe for column descriptions, @source for origin, and @examples for usage, ensuring the dataset is properly documented for users. ```R #' Example Survey Data #' #' Survey responses from 500 participants collected in 2023 for #' demonstrating statistical analysis techniques. Contains demographic #' information and Likert scale responses. #' #' @format A data frame with 500 rows and 8 columns: #' \describe{ #' \item{id}{Participant ID (integer).} #' \item{age}{Age in years (numeric, range 18-65).} #' \item{gender}{Gender identity (factor: "Male", "Female", "Other").} #' \item{response}{Survey response score (numeric, 1-10 scale).} #' \item{group}{Treatment group (factor: "Control", "Treatment").} #' \item{date}{Response date (Date object).} #' } #' @source Simulated data based on typical survey patterns. Generated using `scripts/generate_survey_data.R`. #' @keywords datasets #' @examples #' data(survey_data) #' str(survey_data) #' summary(survey_data$response) #' if (requireNamespace("ggplot2", quietly = TRUE)) { #' ggplot2::ggplot(survey_data, ggplot2::aes(x = age, y = response, color = gender)) + #' ggplot2::geom_point() #' } "survey_data" # The name must match the .rda file (without extension) ``` -------------------------------- ### Convert NeuroVecSeq to HDF5 with Metadata in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/tests/testthat/test_summary_neurovecseq.md This R example demonstrates how to create a NeuroVecSeq object from multiple NeuroVecs with varying time dimensions, and then convert it into an HDF5 file using `neurovecseq_to_h5()`. It showcases specifying scan names, custom chunk dimensions, compression level, and scan-specific metadata for flexible data organization and storage. ```R # Create NeuroVecSeq with different time dimensions vec1 <- NeuroVec(array(data1, dim=c(10,10,5,20)), space1) # 20 time points vec2 <- NeuroVec(array(data2, dim=c(10,10,5,30)), space2) # 30 time points vec3 <- NeuroVec(array(data3, dim=c(10,10,5,25)), space3) # 25 time points nvs <- NeuroVecSeq(vec1, vec2, vec3) # Convert to HDF5 with metadata result <- neurovecseq_to_h5( nvs, file = "output.h5", scan_names = c("rest", "task1", "task2"), chunk_dim = c(10, 10, 10, 50), # Time-optimized compression = 4, scan_metadata = list( rest = list(TR = 2.0, task = "rest"), task1 = list(TR = 2.5, task = "motor"), task2 = list(TR = 2.0, task = "visual") ) ) ``` -------------------------------- ### Example JSON Descriptor for Octree Spatial Transform Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveSpec.md An illustrative example of a non-core, block-based spatial transform described using a JSON descriptor. It defines parameters, associated datasets, inputs, outputs, and capabilities for an `myorg.octree_basis` transform, demonstrating how custom transforms might be structured. ```JSON { "type": "myorg.octree_basis", "version": "0.1", "params": { "depth": 3, "min_components_per_node": 10, "storage_order": "component_x_voxel" }, "datasets": [ { "path": "/plugins/myorg_octree/node_basis_matrices", "role": "basis_collection" }, { "path": "/plugins/myorg_octree/voxel_to_node_map", "role": "partition_map" }, { "path": "/plugins/myorg_octree/node_offset_table", "role": "offset_table" } ], "inputs": ["embedding"], "outputs": ["latent"], "capabilities": { "supports_spatial_subsetting": true, "supports_temporal_subsetting": true }, "schema_uri": "https://myorg.com/schemas/lna/octree_basis_v0.1.schema.json" } ``` -------------------------------- ### APIDOC: Package Structure and Testing Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Outlines the package structure and details the testing strategy, including the use of `driver='core'` and testing various aspects like error handling, multi-run scenarios, checksums, lazy reader behavior, schema validation, default parameter merging, and HDF5 robustness fallbacks. ```APIDOC Package Structure: - Testing: Use `driver='core'`, test error handling, multi-run, checksums, lazy reader, schema validation, default parameter merging, HDF5 robustness fallbacks. ``` -------------------------------- ### Creating Package-Level Documentation File (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Shows the structure of a dedicated R file (e.g., `R/yourpackage-package.R`) for package-level documentation using `roxygen2`. It includes the `_PACKAGE` directive and a placeholder for `usethis` namespace directives. ```R #' @keywords internal "_PACKAGE" ## usethis namespace: start ## (Place @importFrom directives here if desired, or manage them elsewhere) ``` -------------------------------- ### Tips for Efficient hdf5r Implementation Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt This section provides practical tips for optimizing 'hdf5r' implementations, covering consistent voxel ordering, compression trade-offs, chunk size selection, attribute usage, and data validation to enhance performance and reliability. ```APIDOC Consistent Voxel Ordering: - Ensure the order of voxels in voxel_coords matches across datasets for alignment. Compression Trade-offs: - Higher compression levels reduce file size but may impact read/write performance. - Test different levels to find an optimal balance. Chunk Size Selection: - Choose chunk sizes that match typical data access patterns to optimize performance. - For time-series data, chunk along the time dimension if accessing data across time points. Attribute Usage: - Store essential metadata as attributes for quick access. - Use consistent naming conventions for attributes. Data Validation: - Implement checks to ensure data dimensions and types match the specifications. - Use assertions or validation functions during data writing. ``` -------------------------------- ### Generate R Package Documentation Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Commands to generate R package documentation using `roxygen2` and build PDF documentation from Rd files. ```bash Rscript -e "roxygen2::roxygenise()" R CMD Rd2pdf --no-preview . ``` -------------------------------- ### fmristore R Package File Organization Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Outlines the directory structure and purpose of key R files within the `fmristore` package, detailing where S4 classes, generics, constructors, and utility functions are defined. ```APIDOC R/all_class.R: S4 class definitions R/all_generic.R: S4 generic function definitions R/constructors.R: Constructor functions for main classes R/h5_utils.R: HDF5 utility functions (read, write, handle management) R/io_*.R: I/O operations for different formats R/cluster_*.R: Clustered data implementations R/latent_vec.R: Latent representation implementation ``` -------------------------------- ### General hdf5r API Usage Overview Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Overview of fundamental hdf5r functions for creating and managing HDF5 files, groups, datasets, and attributes, including data type definition and compression settings. ```APIDOC H5File$new(filename, mode) file$create_group(group_name) group$create_dataset(name, data, dtype, space, chunk_dims, compression, fill_value) h5attr(dataset, "attribute_name") <- value h5types (built-in types) dataset_create_pl (for compression) ``` -------------------------------- ### Key Design Patterns in fmristore Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Describes the fundamental design patterns implemented in the `fmristore` R package, including HDF5 resource management, S4 generic/method dispatch, lazy evaluation, and the constructor pattern. ```APIDOC 1. HDF5 Resource Management: All HDF5 operations use utility functions in `R/h5_utils.R` that ensure proper handle cleanup via `tryCatch`/`finally` blocks. 2. Generic/Method Dispatch: The package defines S4 generics in `R/all_generic.R` with implementations spread across class-specific files. Key generics include: - h5file(): Access HDF5 file handle - mask(): Access brain mask - basis(): Access spatial basis/components - loadings(): Access temporal loadings - series_concat(): Combine time series data - matrix_concat(): Combine matrix data - as_h5(): Convert to HDF5-backed representation 3. Lazy Evaluation: HDF5-backed classes load data on-demand rather than keeping everything in memory. 4. Constructor Pattern: Each H5-backed class has a constructor function (e.g., `H5NeuroVol()`) that handles HDF5 file validation and object creation. ``` -------------------------------- ### APIDOC: New and Updated Helper Utilities Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Lists and describes new or updated helper functions, such as `lna:::default_params()`, `lna:::has_float16_support()`, `lna::check_transform_implementation()` (with namespace collision warnings), `lna::scaffold_transform()`, and `lna:::schema_cache_clear()`, along with guidance on progress reporting. ```APIDOC New/Updated Helpers: - `lna:::default_params(type)`: Reads schema, extracts defaults. - `lna:::has_float16_support()`: Checks dependencies. - `lna::check_transform_implementation(type)`: Now also warns if `type` namespace collides with core transforms (`quant`, `basis`, `embed`, `temporal`, `delta`) or base R packages (stats, utils, etc.). - `lna::scaffold_transform(type)`: Generates template files, including a stub using `lna:::default_params()`. - `lna:::schema_cache_clear()`: Exposes cache clearing for tests. Progress Reporting: Check `!progressr::handlers_is_empty()` before invoking `progressr`. ``` -------------------------------- ### Run R Package Tests Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Commands to execute `testthat` tests for the R package, either all tests or a specific test file. Demonstrates both shell and R console methods. ```bash Rscript -e "testthat::test_dir('tests/testthat')" Rscript -e "testthat::test_file('tests/testthat/test-h5classes.R')" ``` ```R testthat::test_local() testthat::test_file("tests/testthat/test-cluster_experiment.R") ``` -------------------------------- ### Check and fix R code style with lintr and styler Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Checks for R code style violations using `lintr` and automatically formats code to adhere to the tidyverse style guide using `styler`. Includes an option to preview changes without modifying files. ```R lintr::lint_package() styler::style_pkg() styler::style_pkg(dry = "on") ``` -------------------------------- ### APIDOC: Validation Strategy Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Outlines the validation strategy, covering runtime checks, full audits with checksums, considerations for schema cache in multicore environments, and recommended error reporting practices using `rlang::abort`. ```APIDOC Validation Strategy: - Runtime validation. - Full Audit (`validate_lna`): Includes checksum check if requested. Schema Cache Note: Compiled validators (e.g., from `jsonvalidate`) might not be fork-safe. If using `future::plan(multicore)` heavily involving validation, consider calling `lna:::schema_cache_clear()` within each worker's setup, or use a fork-safe caching strategy if available. - Error Reporting: Use `rlang::abort(..., location = ...)` with step index/name. ``` -------------------------------- ### Documenting Function Parameters with @param (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Shows how to use the `@param` tag in `roxygen2` to document each argument of an R function, including its type and purpose. ```R #' @param x A numeric vector or matrix. #' @param na.rm Logical; if TRUE, NA values are removed before computation. #' @param method Character string specifying computation method: "fast" or "robust". ``` -------------------------------- ### Critical API Usage Guidelines for hdf5r Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt This section outlines critical aspects and best practices for using the 'hdf5r' API, focusing on handling variable-length data, applying compression during dataset creation, and using attributes for metadata storage. ```APIDOC Handling Variable-Length Data: - If using variable-length data types, ensure correct handling with H5T_VLEN$new(). Dataset Creation with Compression: - Apply compression to embedding datasets to optimize storage. Attributes for Basis Reference: - Use attributes to store basis_id, basis_type, and basis_parameters when referencing the basis. ``` -------------------------------- ### APIDOC: `Plan` R6 Class Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Details the `Plan` R6 class, including its public fields like `datasets` (with specific columns and `write_mode` details), and methods such as `add_dataset_def()` which now records `step_index` and sets `write_mode_effective`. ```APIDOC Plan (`plan.R` - R6 Class): Public Fields: - `datasets` (tibble): Columns: `path`, `role`, `producer`, `origin`, `step_index`, `params_json`, `payload_key`, `write_mode` ("eager"/"stream"), `write_mode_effective` (string: actual mode used after fallback, "eager" or potentially "stream" in future). - `descriptors`, `payloads`, `next_index`: (As before) Public Methods: (As before) - `add_dataset_def()`: Now also records `step_index`. `write_mode_effective` is set by `materialise_plan`. - `mark_payload_written()`: (As before) ``` -------------------------------- ### LabeledVolumeSet: Create HDF5 File in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Initializes a new HDF5 file for the LabeledVolumeSet specification using the hdf5r package in write mode. ```R library(hdf5r) file <- H5File$new("labeled_volume_set.h5", mode = "w") ``` -------------------------------- ### LabeledVolumeSet: Critical hdf5r API Usage for Dataset Creation and Attributes Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Details critical hdf5r API calls for creating datasets with compression (chunking and deflate) and assigning attributes to datasets or groups. ```APIDOC Dataset Creation with Compression: H5P_DATASET_CREATE$new() $set_chunk(chunk_dims) $set_deflate(level) Assigning Attributes: h5attr(dataset, "attribute_name") <- value ``` -------------------------------- ### APIDOC: `write_lna` Core Functionality Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Details the responsibilities of the `write_lna` core component, including handling default run naming, resolving transformation parameters, orchestrating the forward pass, and calling the `materialise_plan` function. ```APIDOC write_lna Core (`core_write.R`): - Handles default run naming. - Resolves `transform_params` using specified merge order. - Orchestrates forward pass via `forward_step`. - Calls `materialise_plan`. ``` -------------------------------- ### JSON Descriptor for Physiological Regression Module Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md This JSON configuration defines the `physio_regress` module, specifying its version, parameters such as the regression `method` and `regressor_paths_source`, and detailing the `datasets` it affects or creates, including their `path` patterns and `role`. It also outlines `inputs`, `outputs`, `metadata_updates`, and `capabilities` like `supports_temporal_subsetting`. ```JSON { "type": "physio_regress", "version": "1.0", "params": { "method": "projection", // Could allow other methods like "ICA_removal" later "regressor_paths_source": "runtime", // Indicates regressors provided at write time, not stored in file initially "design_matrix_path_pattern": "/plugins/physio/{run_id}/C", // Pattern for storing C "retain_components": false // Option to store C*pinv*E instead of just C for simpler reversal }, "datasets": [ // Describes datasets *affected* or *created* by this step { "path": "/scans/{run_id}/embedding", // Path pattern - gets resolved by writer loop "role": "embedding", "producer": "physio_regress" // Indicates this transform *modifies* the embedding }, { "path": "/plugins/physio/{run_id}/C", // Path pattern for the stored design matrix "role": "design_matrix", "producer": "physio_regress" // Indicates this transform *creates* the design matrix } // Could add entry for pseudoinverse or projected components if retain_components=TRUE ], "inputs": ["embedding"], // Expects the current embedding from the previous step "outputs": ["embedding"], // Outputs the cleaned (or restored) embedding under the same name "metadata_updates": {}, // Typically no changes to core metadata like TR, dimensions "capabilities": { "supports_spatial_subsetting": false, // Does not affect spatial dimension "supports_temporal_subsetting": true // Regression can be applied to subset of time points }, "schema_uri": "https://neurocompress.org/schemas/lna/2.0/physio_regress.schema.json" // Hypothetical URI } ``` -------------------------------- ### APIDOC: `lna:::materialise_plan` Utility Updates Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Describes updates to the `lna:::materialise_plan` utility, including setting `write_mode_effective`, implementing checksum logic, handling HDF5 errors with retries (compression, chunk size), and using `step_index` for error provenance. ```APIDOC lna:::materialise_plan(...) Updates: - Sets `write_mode_effective` in the `plan` based on actual write behavior (e.g., after fallback). Uses throttled warning for fallback. - Implements checksum logic (write hash attr). - Handles HDF5 errors: - Retry w/o compression on filter errors + warning. - Retry with smaller chunks on write errors + warning (first heuristic: target < 1 GiB; second heuristic if still fails: target <= 256 MiB). - Uses `step_index` for provenance in error messages. ``` -------------------------------- ### Critical Aspects of hdf5r Implementation Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt This section details essential considerations for implementing HDF5 data storage using 'hdf5r', covering data types, compression, chunking, attribute handling, resource management, and advanced features like dataset extensions and variable-length data types. ```APIDOC Data Types and Conversions: - Standard Data Types: Use h5types for predefined HDF5 data types (e.g., h5types$H5T_IEEE_F32LE for 32-bit floats). - Custom Data Types: Create custom data types using classes like H5T_ENUM, H5T_COMPOUND, H5T_ARRAY, and H5T_STRING. - Compound Data Types: Use for datasets that require multiple fields (e.g., cluster metadata). Compression and Chunking: - Compression: - Use Gzip compression via $set_deflate(level) on the dataset creation property list (H5P_DATASET_CREATE). - Level ranges from 0 (no compression) to 9 (maximum compression). - Chunking: - Necessary for compression and extendable datasets. - Set chunk dimensions with $set_chunk(chunk_dims). Handling Attributes: - Assigning Attributes: - Use h5attr(object, "attribute_name") <- value to assign attributes to datasets or groups. - Useful for storing metadata like labels, descriptions, or parameters. Managing HDF5 Resources: - Closing Files and Objects: - Use file$close_all() to close the file and all associated objects. - Important to prevent file locking issues. - Automatic Resource Management: - hdf5r uses R’s garbage collector to manage HDF5 IDs. - Be cautious with open objects to avoid resource leaks. Advanced Features: - Dataset Extensions: - Define maxdims in space to allow dataset resizing. - Use dataset$set_extent(new_dims) to resize datasets if needed. - Variable-Length Data Types: - Use H5T_VLEN$new(dtype_base) for datasets with variable-length entries (e.g., lists). - Enumerated Types: - Use H5T_ENUM$new(labels, values) to define datasets with enumerated types (e.g., cluster IDs). ``` -------------------------------- ### HDF5 Schema Conventions for fmristore Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Details the HDF5 schema conventions followed by the `fmristore` package, including references to specification documents and key HDF5 paths for header, mask, basis, scans, and clusters. ```APIDOC The package follows specific HDF5 schemas documented in `raw-data/`: - BasisEmbeddingSpec.yaml: Latent representation format - ClusteredTimeSeriesSpec.yaml: Clustered voxel data format - LatentNeuroArchiveSpec.md: Comprehensive format specification (LNA 2.0) Key HDF5 paths: - /header/*: NIfTI-compatible header information - /mask: Binary brain mask - /basis/*: Spatial basis/components - /scans//embedding: Temporal coefficients - /clusters/*: Cluster definitions and metadata ``` -------------------------------- ### APIDOC: S3 Dispatch Generics Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Defines the generic functions `forward_step` and `invert_step` used for S3 dispatch within the `lna` package, specifying their parameters. ```APIDOC S3 Dispatch (`dispatch.R`): Generics: - `forward_step(type, desc, handle)` - `invert_step(type, desc, handle)` ``` -------------------------------- ### LabeledVolumeSet: Assign Labels and Metadata to Volumes in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Assigns descriptive labels and additional metadata as attributes to individual volume datasets within the HDF5 file using h5attr. ```R h5attr(volumes_group[["volume_1"]], "label") <- "Anatomical Scan" h5attr(volumes_group[["volume_1"]], "description") <- "T1-weighted MRI" ``` -------------------------------- ### Generate citation for fmristore R package Source: https://github.com/bbuchsbaum/fmristore/blob/main/README.md Displays the recommended citation information for the `fmristore` R package, which can be used when citing the package in academic publications. ```R citation("fmristore") ``` -------------------------------- ### LNA Framework: Handling Complex Processing Requirements Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md This section outlines how the LNA framework and the `lna` package address various data processing requirements. It details mechanisms for per-run operations, auxiliary data storage and retrieval, optional transform reversal, temporal subsetting optimization, and pluggable implementation via S3 methods and JSON Descriptors without altering the core LNA specification. ```APIDOC Requirement: Operate per-run, modify only embeddings How LNA / `lna` Handles It: `forward_step` runs within a run-aware loop (managed by `core_write`); it consumes/updates the `embedding` in the `stash`. Requirement: Store/retrieve auxiliary data (design matrix `C`) How LNA / `lna` Handles It: `forward_step` adds `C` to `handle$plan$payloads`; `materialise_plan` writes it to `/plugins/`. `invert_step` reads from `handle$h5`. Requirement: Optional reversal of the transform at read time How LNA / `lna` Handles It: `invert_step` checks a flag (e.g., passed via `read_lna` options) and conditionally performs the noise restoration logic. Requirement: Optimize for temporal subsetting How LNA / `lna` Handles It: Declares `capabilities.supports_temporal_subsetting = true`. `invert_step` applies subsetting *after* potential restoration. Requirement: No change to core LNA spec or `lna` package How LNA / `lna` Handles It: Implemented entirely via S3 methods and standard JSON Descriptors; fully pluggable. ``` -------------------------------- ### ClusteredTimeSeries: Create HDF5 File in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Initializes a new HDF5 file for the ClusteredTimeSeries specification using the hdf5r package in write mode. ```R file <- H5File$new("clustered_time_series.h5", mode = "w") ``` -------------------------------- ### Adding Function Details with @details (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Demonstrates how to use the `@details` tag in `roxygen2` to provide elaborate explanations, algorithms, or technical specifics for an R function's documentation. ```R #' @details #' Uses Welford's online algorithm for numerically stable computation. #' For matrices, statistics are computed column-wise. ``` -------------------------------- ### APIDOC: Concurrency Support Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md States the concurrency capabilities of the `lna` package, confirming that core read/write operations are safe for basic parallelism. ```APIDOC Concurrency: - Core read/write is safe for basic parallelism. ``` -------------------------------- ### Documenting Function Title and Description (roxygen2 R) Source: https://github.com/bbuchsbaum/fmristore/blob/main/CRAN_guidance.md Illustrates how to define the title and description for an R function's documentation using `roxygen2` comments. The first line serves as the title, followed by a detailed description. ```R #' Compute Summary Statistics #' #' Calculates descriptive statistics for numeric data, handling missing #' values and providing multiple summary measures in a consistent format. ``` -------------------------------- ### Reading and Opening LNA Files in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Provides functions to read LNA (Lazy NIfTI Array) files. `open_lna` is a convenience alias for `read_lna`. ```R read_lna <- function(...) { ... } # Implementation uses core_read ``` ```APIDOC Function: read_lna Description: Reads LNA (Lazy NIfTI Array) files. Parameters: ...: Additional arguments passed to the underlying implementation. Returns: If lazy=TRUE: An object of class `lna_reader`. Exported: Yes ``` ```R open_lna <- read_lna ``` ```APIDOC Function: open_lna Description: Convenience alias for `read_lna`. Parameters: ...: Arguments passed to `read_lna`. Exported: Yes ``` -------------------------------- ### Build All R Vignettes for an R Package Source: https://github.com/bbuchsbaum/fmristore/blob/main/vignettes/README.md This R command uses the `devtools` package to compile all R Markdown vignettes located within the current R package's `vignettes` directory. It generates the corresponding output files (e.g., HTML) for all available vignettes. ```R devtools::build_vignettes() ``` -------------------------------- ### Lint R Package Code Source: https://github.com/bbuchsbaum/fmristore/blob/main/CLAUDE.md Commands to perform static code analysis using `lintr` on the R package, either for the entire package or a specific file. ```bash Rscript -e "lintr::lint_package()" Rscript -e "lintr::lint('R/h5_utils.R')" ``` -------------------------------- ### APIDOC: `read_lna` Core Functionality Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Describes the responsibilities of the `read_lna` core component, such as managing plugin allowances, checking `output_dtype` requirements, orchestrating the reverse pass, and handling lazy return modes. ```APIDOC read_lna Core (`core_read.R`): - Handles `allow_plugins` modes and non-interactive fallback. - Checks `output_dtype == "float16"` requirements. - Orchestrates reverse pass via `invert_step`. - Handles `lazy=TRUE` return. ``` -------------------------------- ### LabeledVolumeSet: Create Header Group and Assign NIfTI-1 Attributes in R Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/hdf5_notes.txt Creates a 'header' group within the HDF5 file and assigns NIfTI-1 header fields as attributes, such as sizeof_hdr and dim, using h5attr. ```R header_group <- file$create_group("header") # Assign NIfTI-1 header fields as attributes h5attr(header_group, "sizeof_hdr") <- 348L h5attr(header_group, "dim") <- c(4L, dim_x, dim_y, dim_z, 1L, 1L, 1L, 1L) # Continue for other header fields... ``` -------------------------------- ### Define API Namespace and Version for myorg.sparsepca Source: https://github.com/bbuchsbaum/fmristore/blob/main/raw-data/LatentNeuroArchiveCodingGuide.md Specifies the namespace and version for the 'myorg.sparsepca' transform, which applies Sparse PCA to input data. This transform can operate on either single-run data or aggregated output. ```APIDOC API Namespace & Version: type: "myorg.sparsepca" version: "1.0" ```