### Inserting Data into Stores

Source: https://context7.com/tidyverse/ragnar/llms.txt

Demonstrates the workflow for reading documents, chunking them, and inserting the chunks into a ragnar data store. This includes examples using multiple web pages.

```APIDOC
## Inserting Data into Stores

### Description
Functions to insert processed text chunks (e.g., from markdown files) into a ragnar data store, making them available for retrieval.

### Complete Workflow: Read, Chunk, Insert
```r
# Create a store configured with OpenAI embeddings
store <- ragnar_store_create(
  "r4ds.duckdb",
  embed = embed_openai(model = "text-embedding-3-small")
)

# List of pages to process
pages <- c(
  "https://r4ds.hadley.nz/data-transform.html",
  "https://r4ds.hadley.nz/data-visualize.html",
  "https://r4ds.hadley.nz/workflow-basics.html"
)

# Loop through pages, read, chunk, and insert
for (page in pages) {
  message("Processing: ", page)
  chunks <- page |> read_as_markdown() |> markdown_chunk()
  ragnar_store_insert(store, chunks)
}
```
```

--------------------------------

### Create and Ingest Knowledge Store with Ragnar (R)

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

This snippet shows how to initialize a Ragnar knowledge store, embed content using OpenAI's text-embedding-3-small model, and ingest markdown content from a list of URLs. It assumes the 'ragnar' and 'stringr' libraries are installed.

```R
library(ragnar)

base_url <- "https://r4ds.hadley.nz"
pages <- ragnar_find_links(base_url)

store_location <- "r4ds.ragnar.duckdb"

store <- ragnar_store_create(
  store_location,
  embed = \(x) ragnar::embed_openai(x, model = "text-embedding-3-small")
)

for (page in pages) {
  message("ingesting: ", page)
  chunks <- page |> read_as_markdown() |> markdown_chunk()
  ragnar_store_insert(store, chunks)
}

ragnar_store_build_index(store)
```

--------------------------------

### Connect to and Retrieve from Knowledge Store (R)

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

This example demonstrates how to connect to an existing Ragnar knowledge store in read-only mode and retrieve relevant text chunks based on a natural language query. It requires the 'ragnar' library.

```R
library(ragnar)
store_location <- "r4ds.ragnar.duckdb"
store <- ragnar_store_connect(store_location, read_only = TRUE)

text <- "How can I subset a dataframe with a logical vector?"

(relevant_chunks <- ragnar_retrieve(store, text))
```

--------------------------------

### Install ragnar Development Version from GitHub

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Installs the development version of the ragnar package directly from GitHub using the 'pak' package manager. This is useful for accessing the latest features or bug fixes before they are released to CRAN.

```r
# install.packages("pak")
pak::pak("tidyverse/ragnar")
```

--------------------------------

### Insert Data into Stores in R

Source: https://context7.com/tidyverse/ragnar/llms.txt

Illustrates a complete RAG workflow: reading multiple web pages, converting them to Markdown, chunking the Markdown content, and inserting the chunks into a Ragnar store. This example uses OpenAI embeddings and DuckDB for storage.

```r
# Complete workflow: read, chunk, insert
store <- ragnar_store_create(
  "r4ds.duckdb",
  embed = embed_openai(model = "text-embedding-3-small")
)

pages <- c(
  "https://r4ds.hadley.nz/data-transform.html",
  "https://r4ds.hadley.nz/data-visualize.html",
  "https://r4ds.hadley.nz/workflow-basics.html"
)

for (page in pages) {
  message("Processing: ", page)
  chunks <- page |> read_as_markdown() |> markdown_chunk()
  ragnar_store_insert(store, chunks)
}

```

--------------------------------

### Install ragnar R Package

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Installs the ragnar package from CRAN. This is the standard way to get the latest stable release of the package for use in R projects.

```r
install.packages("ragnar")
```

--------------------------------

### Chunk Markdown Documents in R

Source: https://context7.com/tidyverse/ragnar/llms.txt

Provides examples of segmenting Markdown content into manageable chunks using `markdown_chunk`. It covers basic chunking with default settings, customization of `target_size` and `target_overlap`, chunking without overlap, and advanced segmentation based on heading levels. Unbounded chunking and accessing the original document are also demonstrated.

```r
# Basic chunking with defaults (1600 characters, 50% overlap)
md <- read_as_markdown("https://r4ds.hadley.nz/data-transform.html")
chunks <- markdown_chunk(md)
print(chunks)
# Shows: start, end, context, text columns

# Customize chunk size and overlap
chunks <- markdown_chunk(md, target_size = 800, target_overlap = 0.3)

# Chunk without overlap
chunks <- markdown_chunk(md, target_size = 1600, target_overlap = 0)

# Segment by heading levels (prevents chunks from crossing these boundaries)
chunks <- markdown_chunk(
  md,
  target_size = 2000,
  segment_by_heading_levels = c(1, 2)
)

# Create unbounded chunks (one per segment)
chunks <- markdown_chunk(
  md,
  target_size = Inf,
  segment_by_heading_levels = c(1, 2)
)

# Access the underlying document
doc <- attr(chunks, "document")
cat(substr(doc, 1, 200))
```

--------------------------------

### Embed with OpenAI's text-embedding-3-small

Source: https://github.com/tidyverse/ragnar/blob/main/tests/testthat/_snaps/store.md

This snippet shows how to use the `ragnar::embed_openai` function to get default embeddings using the 'text-embedding-3-small' model. It assumes the necessary environment variables or configurations for OpenAI are set.

```r
store@embed
function (x) 
ragnar::embed_openai(x = x, model = "text-embedding-3-small")
<environment: base>
```

```r
store@embed
function (x, ...) 
ragnar::embed_openai(x = x, model = "text-embedding-3-small", 
    ...)
<environment: base>
```

--------------------------------

### Subset Rows and Select Columns with Base R

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Shows how to achieve the same row filtering and column selection as the dplyr example, but using only base R functionality with the subset() function. This function operates directly on data frames without external package dependencies.

```r
subset(df, x > 1, select = c(x, y))
```

--------------------------------

### Serve DuckDB Store via MCP Server

Source: https://context7.com/tidyverse/ragnar/llms.txt

Exposes a DuckDB knowledge store over the Model Context Protocol (MCP), making it accessible for external tools and services. Allows for custom preparation and additional tools.

```r
# Serve a store over Model Context Protocol
# This function blocks and runs a server - use in a separate R session
mcp_serve_store(
  store = "r4ds.duckdb",
  store_description = "R for Data Science (2e) book content",
  top_k = 10,
  name = "search_r4ds",
  title = "R4DS Knowledge Base"
)

# With custom preparation and extra tools
my_tools <- list(
  ellmer::tool(
    function(package_name) {
      utils::packageDescription(package_name)$Description
    },
    name = "get_package_info",
    description = "Get description of an R package",
    arguments = list(package_name = ellmer::type_string("R package name"))
  )
)

mcp_serve_store(
  "my_store.duckdb",
  extra_tools = my_tools
)
```

--------------------------------

### Creating and Connecting to Stores

Source: https://context7.com/tidyverse/ragnar/llms.txt

This section details how to create new ragnar data stores, both in-memory and persistent, and how to connect to existing stores with read or write access. It also shows how to configure embedding functions and add custom metadata columns.

```APIDOC
## Creating and Connecting to Stores

### Description
Functions for creating and connecting to ragnar data stores, which utilize DuckDB for storage and support vector similarity search and BM25.

### Creating a Store

#### In-memory Store
```r
store <- ragnar_store_create(location = ":memory:", embed = \(x) ragnar::embed_openai(x, model = "text-embedding-3-small"), embedding_size = 1536)
```

#### Persistent Store on Disk
```r
store <- ragnar_store_create(location = "my_knowledge_base.duckdb", embed = embed_ollama(model = "nomic-embed-text"), overwrite = TRUE)
```

#### Store with Extra Schema Columns
```r
store <- ragnar_store_create(embed = embed_openai(), extra_cols = data.frame(category = character(), source = character(), date = as.Date(character())))
```

### Connecting to a Store

#### Read-only Connection
```r
store <- ragnar_store_connect("my_knowledge_base.duckdb", read_only = TRUE)
```

#### Writable Connection
```r
store <- ragnar_store_connect("my_knowledge_base.duckdb", read_only = FALSE)
```
```

--------------------------------

### Create and Connect to Ragnar Stores

Source: https://context7.com/tidyverse/ragnar/llms.txt

Demonstrates how to create new Ragnar stores, either in-memory or persistent on disk, with various embedding providers (OpenAI, Ollama) and custom schema extensions. It also shows how to connect to existing stores with read-only or read-write access.

```r
library(ragnar)

# Create an in-memory store with OpenAI embeddings
store <- ragnar_store_create(
  location = ":memory:",
  embed = \(x) ragnar::embed_openai(x, model = "text-embedding-3-small"),
  embedding_size = 1536
)

# Create a persistent store on disk
store <- ragnar_store_create(
  location = "my_knowledge_base.duckdb",
  embed = embed_ollama(model = "nomic-embed-text"),
  overwrite = TRUE
)

# Create a store with extra schema columns for metadata
store <- ragnar_store_create(
  embed = embed_openai(),
  extra_cols = data.frame(
    category = character(),
    source = character(),
    date = as.Date(character())
  )
)

# Connect to an existing store (read-only by default)
store <- ragnar_store_connect("my_knowledge_base.duckdb", read_only = TRUE)

# Connect with write access
store <- ragnar_store_connect("my_knowledge_base.duckdb", read_only = FALSE)
```

--------------------------------

### R LLM Chat with Retrieval Tool using ellmer

Source: https://context7.com/tidyverse/ragnar/llms.txt

Sets up an LLM chat session with a system prompt and registers a retrieval tool to access a knowledge store. Allows for conversational querying of the knowledge base.

```r
library(ellmer)

# Set up chat with system prompt
system_prompt <- "
You are an expert R programming assistant. Before responding to questions,
retrieve relevant information from the knowledge store. Always cite sources
with links and quote relevant passages.
"

chat <- ellmer::chat_openai(system_prompt, model = "gpt-4o")

# Register retrieval tool
store <- ragnar_store_connect("r4ds.duckdb", read_only = TRUE)
ragnar_register_tool_retrieve(chat, store, top_k = 10)

# Chat with retrieval
response <- chat$chat("How can I filter a dataframe to only include certain rows?")
cat(response)

# Multiple queries in a conversation
chat$chat("What about joining two dataframes together?")
chat$chat("Can you show me an example using the dplyr package?")

# Inspect chat history
chat$turns()
```

--------------------------------

### Register Retrieve Tool for ellmer Chat with ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Equips an 'ellmer::Chat' object with a retrieval tool, enabling the LLM to fetch relevant context from a store on demand. This integrates retrieval capabilities directly into conversational AI.

```r
ragnar_register_tool_retrieve(chat, store)
```

--------------------------------

### Download TailwindCSS CLI for macOS arm64

Source: https://github.com/tidyverse/ragnar/blob/main/inst/store-inspector/README.md

This snippet provides the command to download the TailwindCSS command-line interface for macOS on ARM architecture. It ensures the executable has the necessary permissions to run.

```shell
# Example for macOS arm64
curl -sLO https://github.com/tailwindlabs/tailwindcss/releases/latest/download/tailwindcss-macos-arm64
chmod +x tailwindcss-macos-arm64
mv tailwindcss-macos-arm64 tailwindcss
```

--------------------------------

### Interactive Store Inspection with Shiny

Source: https://context7.com/tidyverse/ragnar/llms.txt

Launches a Shiny application to browse the contents of a ragnar knowledge store. Provides an interactive interface for searching, previewing chunks, and visualizing embeddings.

```r
# Launch Shiny app to browse store contents
store <- ragnar_store_connect("r4ds.duckdb")
ragnar_store_inspect(store)

# Search and preview chunks interactively
# - Type queries to test VSS/BM25 search
# - Click documents to see full text and metadata
# - Keyboard shortcuts: / to focus search, j/k to navigate

# Visualize embeddings with Embedding Atlas (requires Python packages)
# Requires: pip install embedding-atlas duckdb==<matching-version>
ragnar_store_atlas(
  store,
  host = "localhost",
  port = 3030,
  launch.browser = TRUE
)
```

--------------------------------

### Create and Connect to a ragnar DuckDB Store

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Initializes and connects to a DuckDB database for storing processed data. This is the default storage mechanism in ragnar, optimized for efficient RAG operations.

```r
ragnar_store_create()
ragnar_store_connect()
```

--------------------------------

### Concurrent Ingestion with Multiple Workers in Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Demonstrates parallel ingestion of multiple documents into a Ragnar store. It allows specifying the number of workers and provides a progress indicator. Includes functions for finding links on websites and preparing data for ingestion.

```r
# Ingest many documents in parallel
store <- ragnar_store_create(
  "large_knowledge_base.duckdb",
  embed = embed_openai(model = "text-embedding-3-small")
)

# Find all links on a website
pages <- ragnar_find_links("https://r4ds.hadley.nz", depth = 0)

# Ingest with parallel workers (default: auto-detect cores)
ragnar_store_ingest(
  store,
  paths = pages,
  prepare = \(path) {
    path |> \n      read_as_markdown() |> \n      markdown_chunk(target_size = 1600, target_overlap = 0.5)
  },
  n_workers = 4,
  progress = TRUE
)

ragnar_store_build_index(store)

# Custom preparation function with error handling
ragnar_store_ingest(
  store,
  paths = c("doc1.pdf", "doc2.html", "doc3.md"),
  prepare = \(path) {
    tryCatch({
      md <- read_as_markdown(path)
      chunks <- markdown_chunk(md, target_size = 2000)
      chunks$source_type <- tools::file_ext(path)
      chunks
    }, error = function(e) {
      warning("Failed to process ", path, ": ", e$message)
      NULL
    })
  }
)
```

--------------------------------

### Register Ragnar Retrieve Tool for LLM Interaction (R)

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

This code illustrates how to register the Ragnar retrieval tool with an OpenAI chat model. This allows the LLM to query the knowledge store before generating a response, enhancing its ability to provide contextually relevant answers. It utilizes the 'ragnar', 'stringr', and 'ellmer' libraries.

```R
library(ragnar)
store_location <- "r4ds.ragnar.duckdb"
store <- ragnar_store_connect(store_location, read_only = TRUE)

system_prompt <- stringr::str_squish(
  "\n  You are an expert R programmer and mentor. You are concise.\n\n  Before responding, retrieve relevant material from the knowledge store. Quote or\n  paraphrase passages, clearly marking your own words versus the source. Provide a\n  working link for every source cited, as well as any additional relevant links.\n  Do not answer unless you have retrieved and cited a source.\n  "
)
chat <- ellmer::chat_openai(
  system_prompt,
  model = "gpt-4.1"
)

ragnar_register_tool_retrieve(chat, store, top_k = 10)

chat$chat("How can I subset a dataframe?")
```

--------------------------------

### Compile TailwindCSS for Ragnar App

Source: https://github.com/tidyverse/ragnar/blob/main/inst/store-inspector/README.md

This command compiles the main CSS file (`www/app.css`) into the output file (`www/app.out.css`) using the TailwindCLI. This is essential for applying the generated styles to the Shiny app.

```shell
./tailwindcss -i www/app.css -o www/app.out.css
```

--------------------------------

### Converting Documents to Markdown

Source: https://context7.com/tidyverse/ragnar/llms.txt

Learn how to convert various document formats (like PDF and HTML) into markdown using the `read_as_markdown` function. This includes options for specifying CSS selectors and tracking the origin of the content.

```APIDOC
## Converting Documents to Markdown

### Description
Functions to convert local files (e.g., PDF) or web pages into markdown format for further processing.

### Converting a Local PDF File
```r
pdf_path <- file.path(R.home("doc"), "NEWS.pdf")
md <- read_as_markdown(pdf_path)
```

### Converting a Webpage
```r
md <- read_as_markdown("https://r4ds.hadley.nz/base-R.html")
```

### Extracting Specific Content using CSS Selectors
```r
md <- read_as_markdown(
  "https://quarto.org/docs/computations/python.html",
  html_extract_selectors = "main",
  html_zap_selectors = c("#quarto-sidebar", "header", "footer", "nav")
)
```

### Converting with Origin Tracking
```r
md <- read_as_markdown(
  "https://r4ds.hadley.nz/base-R.html",
  origin = "https://r4ds.hadley.nz/base-R.html"
)
# The origin is stored as an attribute
attr(md, "origin")
```
```

--------------------------------

### Insert Markdown Chunks into Ragnar Store

Source: https://context7.com/tidyverse/ragnar/llms.txt

Reads a markdown file, splits it into manageable chunks, assigns metadata (category and priority), and inserts these chunks into the Ragnar store.

```r
md <- read_as_markdown("https://r4ds.hadley.nz/base-R.html")
chunks <- markdown_chunk(md)
chunks$category <- "tutorial"
chunks$priority <- 1L
ragnar_store_insert(store, chunks)
```

--------------------------------

### Convert Document to Markdown using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Processes various document types into Markdown format using the MarkItDown library. This is the first step in preparing documents for RAG workflows, enabling consistent text processing.

```r
read_as_markdown()
```

--------------------------------

### Embedding Text

Source: https://context7.com/tidyverse/ragnar/llms.txt

This section explains how to generate text embeddings using various providers like Ollama and OpenAI. It demonstrates creating reusable embedding functions and embedding dataframes.

```APIDOC
## Embedding Text

### Description
Functions to generate numerical vector representations (embeddings) for text using different embedding models and providers.

### Embedding with Ollama (Local)
```r
text <- c("R is a programming language", "Python is also popular")
embeddings <- embed_ollama(text, model = "nomic-embed-text")
dim(embeddings) # Returns dimensions of the embedding matrix
```

### Embedding with OpenAI
```r
embeddings <- embed_openai(
  text,
  model = "text-embedding-3-small",
  api_key = Sys.getenv("OPENAI_API_KEY")
)
dim(embeddings)
```

### Creating a Reusable Embedding Function
```r
embed_fn <- embed_ollama(model = "nomic-embed-text", base_url = "http://localhost:11434")
embeddings <- embed_fn(text)
```

### Embedding with Batch Size Control
```r
embeddings <- embed_openai(text, batch_size = 50)
```

### Embedding a Dataframe
```r
df <- data.frame(text = c("first chunk", "second chunk"))
df <- embed_openai(df) # Adds an 'embedding' column to the dataframe
str(df$embedding)
```
```

--------------------------------

### Create Ragnar Store with Embeddings and Metadata

Source: https://context7.com/tidyverse/ragnar/llms.txt

Creates a new Ragnar store, optionally configuring embedding functions (e.g., Ollama) and specifying extra metadata columns for ingested data.

```r
store <- ragnar_store_create(
  embed = embed_ollama(),
  extra_cols = data.frame(category = character(), priority = integer())
)
```

--------------------------------

### Convert Documents to Markdown in R

Source: https://context7.com/tidyverse/ragnar/llms.txt

Shows how to convert various document sources, including local PDF files and web pages, into Markdown format using `read_as_markdown`. It also covers advanced usage like extracting specific HTML content via CSS selectors and tracking the origin of the converted content.

```r
# Convert a local PDF file to markdown
pdf_path <- file.path(R.home("doc"), "NEWS.pdf")
md <- read_as_markdown(pdf_path)
cat(strsplit(md, "\n")[[1]][1:10], sep = "\n")

# Convert a webpage to markdown
md <- read_as_markdown("https://r4ds.hadley.nz/base-R.html")

# Extract specific content using CSS selectors
md <- read_as_markdown(
  "https://quarto.org/docs/computations/python.html",
  html_extract_selectors = "main",
  html_zap_selectors = c("#quarto-sidebar", "header", "footer", "nav")
)

# Convert with origin tracking
md <- read_as_markdown(
  "https://r4ds.hadley.nz/base-R.html",
  origin = "https://r4ds.hadley.nz/base-R.html"
)
# The origin is stored as an attribute
attr(md, "origin")
```

--------------------------------

### Retrieve Content using Vector Similarity Search (VSS) with ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Retrieves text chunks using vector similarity search via the DuckDB VSS extension. This method is effective for finding semantically similar content based on embeddings.

```r
ragnar_retrieve_vss()
```

--------------------------------

### View and Query Ragnar Chunks

Source: https://context7.com/tidyverse/ragnar/llms.txt

Allows viewing text chunks before insertion into the store and querying the store directly using dplyr. Supports different store versions with distinct table structures.

```r
# View chunks before inserting (useful for iterating on chunking strategy)
md <- read_as_markdown("https://r4ds.hadley.nz/base-R.html")
chunks <- markdown_chunk(md, target_size = 1000, target_overlap = 0.4)
ragnar_chunks_view(chunks)

# Query store directly with dplyr
library(dplyr)
store <- ragnar_store_connect("r4ds.duckdb")

# Version 1 stores: single 'chunks' table
store |>
  dplyr::tbl("chunks") |>
  select(origin, text) |>
  head() |>
  collect()

# Version 2 stores: 'documents' and 'embeddings' tables
store |> #
  dplyr::tbl("documents") |>
  select(doc_id, origin, text) |>
  head() |>
  collect()

store |> #
  dplyr::tbl("embeddings") |>
  select(doc_id, start, end, context) |>
  head() |>
  collect()
```

--------------------------------

### Embed Text Using Various Providers in R

Source: https://context7.com/tidyverse/ragnar/llms.txt

Demonstrates text embedding using different providers like Ollama and OpenAI. It shows how to embed character vectors, create reusable embedding functions, control batch sizes, and embed entire dataframes, adding an 'embedding' column to the dataframe.

```r
# Embed with Ollama (local)
text <- c("R is a programming language", "Python is also popular")
embeddings <- embed_ollama(text, model = "nomic-embed-text")
dim(embeddings)  # [1] 2 768

# Embed with OpenAI
embeddings <- embed_openai(
  text,
  model = "text-embedding-3-small",
  api_key = Sys.getenv("OPENAI_API_KEY")
)
dim(embeddings)  # [1] 2 1536

# Create a reusable embedding function
embed_fn <- embed_ollama(model = "nomic-embed-text", base_url = "http://localhost:11434")
embeddings <- embed_fn(text)

# Embed with batch size control
embeddings <- embed_openai(text, batch_size = 50)

# Embed dataframe (adds embedding column)
df <- data.frame(text = c("first chunk", "second chunk"))
df <- embed_openai(df)
str(df$embedding)
```

--------------------------------

### Build Search Index with Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Builds a search index for a given store, which is a prerequisite for efficient data retrieval. Supports both 'vss' (Vector Similarity Search) and 'fts' (Full-Text Search) index types.

```r
ragnar_store_build_index(store, type = c("vss", "fts"))
```

--------------------------------

### Ragnar R Package Update Function

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

This R code demonstrates how to update the RAG index using the 'ragnar' package. It specifies the database file, reads HTML documents, defines text chunking strategy (semantic splitting with size and overlap), and sets the embedding model to OpenAI's 'text-embedding-3-small'.

```r
library(ragnar)

ragnar_update(
  db = "myproject.duckdb",
  documents = ragnar_read_html("path/to/docs"),
  chunk = split_semantic(chunk_size = 500, chunk_overlap = 50),
  embedding = embed_openai(model = "text-embedding-3-small")
)

# Nice to have some iterator API especially for experimenting
```

--------------------------------

### Chunk Markdown Text using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Divides Markdown text into semantically meaningful chunks. This function offers intelligent chunking strategies and associates each chunk with relevant document headings, aiding context augmentation.

```r
markdown_chunk()
```

--------------------------------

### Subset Rows and Select Columns with dplyr in R

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Demonstrates how to filter rows based on a condition and select specific columns from a data frame using the dplyr package. Requires the dplyr library to be loaded. Takes a data frame as input and returns a subsetted data frame.

```r
library(dplyr)
df %>% filter(x > 1) %>% select(x, y)
```

--------------------------------

### Watch and Recompile TailwindCSS for Ragnar App

Source: https://github.com/tidyverse/ragnar/blob/main/inst/store-inspector/README.md

This command compiles the CSS file and enables a file watcher. The TailwindCLI will automatically recompile `www/app.out.css` whenever changes are detected in `www/app.css` or other relevant source files during development.

```shell
./tailwindcss -i www/app.css -o www/app.out.css --watch
```

--------------------------------

### Retrieve Data Chunks from DuckDB using Ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

Retrieves data chunks from a specified DuckDB database using a SQL query. It takes the database path and the query string as input and returns the matching data chunks.

```R
chunks <- ragnar_retrieve(
  db = "myproject.duckdb",
  query = "foo")
```

--------------------------------

### Embed with Amazon Bedrock titan-embed-text-v1

Source: https://github.com/tidyverse/ragnar/blob/main/tests/testthat/_snaps/store.md

This snippet illustrates the usage of `ragnar::embed_bedrock` for generating embeddings using Amazon Bedrock's 'amazon.titan-embed-text-v1' model. It requires AWS credentials and a 'default' profile to be configured.

```r
store@embed
function (x) 
{
    ragnar::embed_bedrock(x = x, model = "amazon.titan-embed-text-v1", 
        profile = "default")
}
<environment: base>
```

--------------------------------

### Create DuckDB Table for Chunks

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

This SQL snippet defines the schema for the 'chunks' table in a DuckDB database. It includes columns for chunk identification, file origin, content, embedding vectors, and token count, essential for storing processed document segments.

```sql
CREATE TABLE chunks (
    chunk_id INTEGER PRIMARY KEY,
    file_path TEXT,
    file_hash TEXT,
    chunk_index INTEGER,
    content TEXT,
    embedding INTEGER[128],
    token_count INTEGER
);
```

--------------------------------

### Embed with Ollama embeddinggemma:300m

Source: https://github.com/tidyverse/ragnar/blob/main/tests/testthat/_snaps/store.md

This snippet demonstrates embedding text using the `ragnar::embed_ollama` function with the 'embeddinggemma:300m' model from Ollama. Ensure Ollama is running and the specified model is available.

```r
store@embed
function (x) 
ragnar::embed_ollama(x = x, model = "embeddinggemma:300m")
<environment: base>
```

--------------------------------

### Chunking Markdown Documents

Source: https://context7.com/tidyverse/ragnar/llms.txt

This section covers the `markdown_chunk` function for segmenting markdown text into smaller, manageable chunks. Options for controlling chunk size, overlap, and segmentation by heading levels are explained.

```APIDOC
## Chunking Markdown Documents

### Description
Functions to segment markdown text into smaller chunks suitable for embedding and retrieval.

### Basic Chunking (Defaults: 1600 characters, 50% overlap)
```r
md <- read_as_markdown("https://r4ds.hadley.nz/data-transform.html")
chunks <- markdown_chunk(md)
print(chunks) # Shows: start, end, context, text columns
```

### Customizing Chunk Size and Overlap
```r
chunks <- markdown_chunk(md, target_size = 800, target_overlap = 0.3)
```

### Chunking Without Overlap
```r
chunks <- markdown_chunk(md, target_size = 1600, target_overlap = 0)
```

### Segmenting by Heading Levels
```r
chunks <- markdown_chunk(
  md,
  target_size = 2000,
  segment_by_heading_levels = c(1, 2) # Prevents chunks from crossing heading levels 1 and 2
)
```

### Creating Unbounded Chunks (One per Segment)
```r
chunks <- markdown_chunk(
  md,
  target_size = Inf, # Sets chunk size to infinity, effectively creating one chunk per segment
  segment_by_heading_levels = c(1, 2)
)
```

### Accessing the Underlying Document
```r
doc <- attr(chunks, "document")
cat(substr(doc, 1, 200))
```
```

--------------------------------

### Retrieve Content using BM25 Text Search with ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Retrieves text chunks using BM25 (sparse vector) text search via the DuckDB full-text search extension. This approach is useful for keyword-based relevance ranking.

```r
ragnar_retrieve_bm25()
```

--------------------------------

### Arrange and Rerank Data Chunks with Ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

Arranges retrieved data chunks using a reranking model, specifically `rerank_voyager`. It allows specifying the model and the number of top results (`top_k`) to consider for reranking.

```R
chunks <- ragnar_arrange(chunks, rerank_voyager(model = "rerank-2", top_k = 10))
```

--------------------------------

### Retrieve Content using ragnar (Vector Similarity and BM25)

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Performs retrieval of relevant text chunks based on a given prompt. This high-level function combines vector similarity search (VSS) and BM25 text search, followed by de-overlapping results.

```r
ragnar_retrieve()
```

--------------------------------

### BM25 Full-Text Search in Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Executes BM25 full-text search queries against the store. Allows adjustment of BM25 parameters (k, b), supports conjunctive searches, and can combine search with metadata filtering.

```r
# Basic BM25 search
results <- ragnar_retrieve_bm25(
  store,
  "filter dataframe rows",
  top_k = 5
)

# Adjust BM25 parameters (k controls term frequency saturation, b controls length normalization)
results <- ragnar_retrieve_bm25(
  store,
  "ggplot2 visualization",
  top_k = 5,
  k = 1.5,
  b = 0.8
)

# Conjunctive search (all terms must be present)
results <- ragnar_retrieve_bm25(
  store,
  "filter AND mutate AND dplyr",
  top_k = 5,
  conjunctive = TRUE
)

# Combine with filters
results <- ragnar_retrieve_bm25(
  store,
  "machine learning",
  top_k = 10,
  filter = source == "advanced_chapters"
)

# Examine BM25 scores (higher = more relevant)
print(results[, c("text", "bm25")])
```

--------------------------------

### Finding Links on Webpages with Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Provides functionality to find links within web pages. It supports recursive searching, filtering child links, and custom URL filtering. It can also validate URL accessibility.

```r
# Find all links on a single page
links <- ragnar_find_links("https://r4ds.hadley.nz")
head(links, 10)

# Find links recursively (depth = 1 follows one level of child links)
links <- ragnar_find_links(
  "https://ellmer.tidyverse.org/",
  depth = 1,
  children_only = TRUE,
  progress = TRUE
)

# Filter to only child links of a specific prefix
links <- ragnar_find_links(
  "https://github.com/user/repo/tree/main/docs",
  children_only = "https://github.com/user/repo",
  depth = 1
)

# Custom URL filtering
links <- ragnar_find_links(
  "https://r4ds.hadley.nz",
  url_filter = \(urls) {
    # Only keep chapter pages
    grep("/[a-z-]+\.html$", urls, value = TRUE)
  }
)

# Validate that URLs are accessible
links <- ragnar_find_links(
  "https://r4ds.hadley.nz",
  validate = TRUE
)
# Check problems if any occurred
if (!is.null(attr(links, "problems"))) {
  print(attr(links, "problems"))
}
```

--------------------------------

### Combined Retrieval (VSS + BM25) in Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Performs a hybrid search by combining both VSS and BM25 retrieval methods. Allows disabling deoverlapping and applying metadata filters to the combined results. Shows how to access scores and metadata from the results.

```r
# Hybrid search combines both methods
results <- ragnar_retrieve(store, "How to create plots with ggplot2?", top_k = 5)

# Results contain scores from both methods
print(names(results))
# [1] "origin" "doc_id" "chunk_id" "start" "end"
# [6] "cosine_distance" "bm25" "context" "text"

# Disable deoverlapping (version 2 stores)
results <- ragnar_retrieve(
  store,
  "data manipulation",
  top_k = 10,
  deoverlap = FALSE
)

# With filtering
results <- ragnar_retrieve(
  store,
  "statistical modeling",
  top_k = 8,
  filter = category %in% c("statistics", "modeling")
)

# Access chunk metadata
print(results[1, c("origin", "context", "text")])
```

--------------------------------

### Compute Embeddings with OpenAI using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Generates text embeddings using the OpenAI API. This function integrates with popular LLM providers for creating vector representations of text chunks required for retrieval.

```r
embed_openai()
```

--------------------------------

### Deoverlapping Retrieved Chunks in Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Demonstrates how to retrieve chunks from the store while controlling the deoverlapping process. Setting `deoverlap = FALSE` retrieves all overlapping chunks, potentially increasing the number of results.

```r
# Retrieve with overlapping chunks
results <- ragnar_retrieve(store, "data transformation", top_k = 15, deoverlap = FALSE)
row(results)  # e.g., 15 rows
```

--------------------------------

### Compute Embeddings with AWS Bedrock using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Generates text embeddings using AWS Bedrock. This function provides integration with cloud-based embedding services, expanding the options for vectorizing text data.

```r
embed_bedrock()
```

--------------------------------

### Chunk Documents with Ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

Iterates through documents provided by the `ragnar_chunk` function, processing them in manageable chunks. This is useful for large datasets that need to be processed sequentially.

```R
for(chunk in ragnar_chunk(documents)) {
  ...
}
```

--------------------------------

### Compute Embeddings with Databricks using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Generates text embeddings using Databricks models. This function allows users to leverage Databricks for embedding generation within their RAG workflows.

```r
embed_databricks()
```

--------------------------------

### Vector Similarity Search (VSS) Retrieval in Ragnar

Source: https://context7.com/tidyverse/ragnar/llms.txt

Performs Vector Similarity Search (VSS) to retrieve relevant documents based on a query. Supports different similarity methods (cosine, euclidean), custom query vectors, and filtering results by metadata.

```r
# Connect to store
store <- ragnar_store_connect("r4ds.duckdb", read_only = TRUE)

# Basic VSS retrieval (default: top 3, cosine distance)
query <- "How do I filter rows in a dataframe?"
results <- ragnar_retrieve_vss(store, query, top_k = 5)
print(results[, c("origin", "text")])

# Use different similarity methods
results <- ragnar_retrieve_vss(
  store,
  query,
  top_k = 5,
  method = "euclidean_distance"
)

# Use custom query vector
my_embedding <- embed_openai(query)
results <- ragnar_retrieve_vss(
  store,
  query,
  query_vector = my_embedding,
  top_k = 10
)

# Filter results by metadata
results <- ragnar_retrieve_vss(
  store,
  "data visualization",
  top_k = 10,
  filter = category == "tutorial" & priority > 5
)

# Examine similarity scores
print(results[, c("text", "cosine_distance")])
```

--------------------------------

### Compute Embeddings with Google Vertex AI using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Generates text embeddings using Google Vertex AI. This function enables the use of Google's powerful AI platform for creating embeddings for RAG applications.

```r
embed_google_vertex()
```

--------------------------------

### Find Links in Webpage using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Extracts all hyperlinks from a given webpage. This function is useful for gathering relevant URLs from web content as part of the document processing stage.

```r
ragnar_find_links()
```

--------------------------------

### Insert Data into ragnar Store

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Adds processed data, including text chunks and their embeddings, into the ragnar storage. This function is crucial for populating the RAG system with searchable information.

```r
ragnar_store_insert()
```

--------------------------------

### Slice Data Chunks by Max Tokens using Ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/vignettes/articles/spec.md

Slices a collection of data chunks to a maximum number of chunks (`n`) while also enforcing a maximum token limit (`max_tokens`) per chunk. This is useful for controlling the size of data passed to language models.

```R
chunks <- rangar_slice_max(chunks, n = 10, max_tokens = 2000)
```

--------------------------------

### Compute Embeddings with Ollama using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Generates text embeddings using the Ollama service. This function is part of the embedding step in RAG, allowing for vector representations of text chunks compatible with similarity search.

```r
embed_ollama()
```

--------------------------------

### Manual Deoverlapping of Chunks

Source: https://context7.com/tidyverse/ragnar/llms.txt

Merges overlapping text ranges within the same document to create cleaner, non-overlapping segments for retrieval. This process helps in refining the ingested data for better accuracy.

```r
# Manual deoverlapping (merges overlapping ranges from same document)
deoverlapped <- chunks_deoverlap(store, results)
nrow(deoverlapped)  # e.g., 8 rows (overlaps merged)

# Compare original vs deoverlapped
print(results[1:3, c("start", "end", "text")])
print(deoverlapped[1, c("start", "end", "text")])
```

--------------------------------

### De-overlap Retrieved Chunks using ragnar

Source: https://github.com/tidyverse/ragnar/blob/main/README.md

Consolidates and removes overlapping text chunks from retrieval results. This function ensures that the final set of retrieved chunks is concise and non-redundant.

```r
chunks_deoverlap()
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.