### Install tidytext Package Development Version from GitHub

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This code snippet demonstrates how to install the development version of the tidytext R package directly from GitHub using the remotes package. This is useful for accessing the latest features or bug fixes.

```r
library(remotes)
install_github("juliasilge/tidytext")
```

--------------------------------

### Install tidytext Package from CRAN

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This code snippet shows how to install the tidytext R package from the Comprehensive R Archive Network (CRAN). This is the standard method for installing stable versions of R packages.

```r
install.packages("tidytext")
```

--------------------------------

### tidylda Installation Failure

Source: https://github.com/juliasilge/tidytext/blob/main/revdep/failures.md

This snippet indicates a general installation failure for the 'tidylda' R package. The specific reason for the failure is not detailed here, but it suggests referring to the installation log file for more information.

```R
* checking whether package ‘tidylda’ can be installed ... ERROR
    Installation failed.
    See ‘/Users/juliasilge/Work/runconf16/tidytext/revdep/checks.noindex/tidylda/new/tidylda.Rcheck/00install.out’ for details.
```

--------------------------------

### oRus Installation Error: Java Architecture Mismatch

Source: https://github.com/juliasilge/tidytext/blob/main/revdep/failures.md

This snippet shows the error encountered during the installation of the 'oRus' R package. The failure is due to a Java runtime environment (JRE) architecture mismatch, where the system expects 'arm64' or 'arm64e' but finds an 'x86_64' compatible library.

```R
** installing *source* package ‘oRus’ ...
** this is package ‘oRus’ version ‘1.0.0’
** package ‘oRus’ successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
Error: .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(jli, FALSE)
  error: unable to load shared object '/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/jli/libjli.dylib':
  dlopen(/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/jli/libjli.dylib, 0x000A): tried: '/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/jli/libjli.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/jli/libjli.dylib' (no such file), '/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre/lib/jli/libjli.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))
Execution halted
ERROR: lazy loading failed for package ‘oRus’
* removing ‘/Users/juliasilge/Work/runconf16/tidytext/revdep/checks.noindex/oRus/new/oRus.Rcheck/oRus’
```

--------------------------------

### C++ Compilation and Linker Command Failure

Source: https://github.com/juliasilge/tidytext/blob/main/revdep/failures.md

This snippet represents the terminal output during a failed R package installation. It highlights linker errors where specific libraries like emutls_w are missing during the shared object creation process.

```bash
clang++ -arch arm64 -std=gnu++17 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o mvrsquared.so RcppExports.o calc_sum_squares_latent.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0 -L/opt/gfortran/lib -lemutls_w -lheapt_w -lgfortran -lquadmath -lpthread -F/Library/Frameworks/R.framework/.. -framework R
ld: warning: search path '/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/14.2.0' not found
ld: warning: search path '/opt/gfortran/lib' not found
ld: library 'emutls_w' not found
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mvrsquared.so] Error 1
```

--------------------------------

### Create Quanteda Document-Feature Matrix (DFM)

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts tidy data frames into a dfm object for use with the quanteda package. Requires tidy data with columns for document, term, and value. The quanteda package must be installed.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)
# library(quanteda)

book_words <- austen_books() |
  unnest_tokens(word, text) |
  count(book, word)

# Cast to quanteda dfm (requires quanteda package)
# book_dfm <- cast_dfm(book_words, document = book, term = word, value = n)
# Returns: Document-feature matrix of: 6 documents, 14,520 features
```

--------------------------------

### Parts of Speech Tagging with parts_of_speech Dataset

Source: https://context7.com/juliasilge/tidytext/llms.txt

Demonstrates how to join the parts_of_speech dataset with tokenized text to analyze the distribution of nouns, verbs, and other grammatical categories.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

austen_books() |>
  unnest_tokens(word, text) |>
  inner_join(parts_of_speech, by = "word") |>
  count(pos, sort = TRUE)
```

--------------------------------

### Load and Prepare Jane Austen Books Data (R)

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Loads Jane Austen's books from the 'janeaustenr' package and restructures them into a format grouped by book and line number. This is a prerequisite for tokenization.

```r
library(janeaustenr)
library(dplyr)

original_books <- austen_books() |
  group_by(book) |
  mutate(line = row_number()) |
  ungroup()

original_books
```

--------------------------------

### Tokenize by Structure (Sentences, Lines, Paragraphs)

Source: https://context7.com/juliasilge/tidytext/llms.txt

Convenience wrappers for splitting text into structural units like sentences, lines, or paragraphs.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)

d <- tibble(txt = prideprejudice)

d |>
  unnest_sentences(sentence, txt)

d |>
  unnest_sentences(sentence, txt, strip_punct = TRUE)

d |>
  unnest_lines(line, txt)

d |>
  unnest_paragraphs(paragraph, txt, paragraph_break = "\n\n")
```

--------------------------------

### Tidy Structural Topic Models (STM)

Source: https://context7.com/juliasilge/tidytext/llms.txt

Provides tidiers for Structural Topic Models (STM) from the stm package. Supports extracting and tidying beta (word-topic), gamma/theta (document-topic), FREX, and lift matrices, as well as model summaries (glance).

```r
library(tidytext)
library(dplyr)
library(stm)
library(janeaustenr)
library(ggplot2)

# Prepare data for STM
austen_sparse <- austen_books() |
  unnest_tokens(word, text) |
  anti_join(stop_words) |
  count(book, word) |
  cast_sparse(book, word, n)

# Fit STM model
topic_model <- stm(austen_sparse, K = 6, verbose = FALSE)

# Tidy word-topic probabilities (beta)
td_beta <- tidy(topic_model, matrix = "beta")
td_beta |
  group_by(topic) |
  slice_max(beta, n = 10)

# High FREX words (frequent and exclusive)
tidy(topic_model, matrix = "frex")

# High lift words
tidy(topic_model, matrix = "lift")

# Document-topic probabilities (gamma/theta)
td_gamma <- tidy(topic_model, matrix = "gamma",
                  document_names = rownames(austen_sparse))
td_gamma

# Model summary
glance(topic_model)
# A tibble with: k, docs, terms, iter, alpha
```

--------------------------------

### cast_sparse - Create Sparse Matrices

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts tidy data frames into sparse matrix formats suitable for machine learning algorithms.

```APIDOC
## cast_sparse

### Description
Converts tidy data into a sparse matrix format, useful for machine learning algorithms.

### Parameters
- **data** (data.frame) - Required - The tidy data frame.
- **row** (column) - Required - The column to be used as matrix rows.
- **column** (column) - Required - The column to be used as matrix columns.
- **value** (column) - Optional - The column containing values for the matrix cells.

### Response
- **matrix** (dgCMatrix) - A sparse matrix representation of the input data.
```

--------------------------------

### Tokenize with Regular Expressions

Source: https://context7.com/juliasilge/tidytext/llms.txt

Splits text into tokens based on custom regular expression patterns.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)

d <- tibble(txt = prideprejudice)

d |>
  unnest_regex(chapter, txt, pattern = "Chapter [\\d]")

d |>
  unnest_regex(word, txt, pattern = "\\s+")

d |>
  unnest_regex(segment, txt, pattern = ",\\s*")
```

--------------------------------

### Tidy LDA and CTM Topic Models

Source: https://context7.com/juliasilge/tidytext/llms.txt

Extracts and tidies results from Latent Dirichlet Allocation (LDA) and Correlated Topic Models (CTM) fitted using the topicmodels package. Supports tidying beta (word-topic), gamma (document-topic) matrices, and model summaries (glance).

```r
library(tidytext)
library(dplyr)
library(topicmodels)
library(ggplot2)

# Load and fit LDA model
data("AssociatedPress", package = "topicmodels")
ap <- AssociatedPress[1:100, ]
lda_model <- LDA(ap, k = 4, control = list(seed = 1234))

# Tidy beta matrix (word-topic probabilities)
td_beta <- tidy(lda_model, matrix = "beta")
td_beta
# A tibble with columns: topic, term, beta

# Find top words per topic
top_terms <- td_beta |
  group_by(topic) |
  slice_max(beta, n = 10) |
  ungroup()

# Visualize topics
ggplot(top_terms, aes(beta, reorder(term, beta), fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free")

# Tidy gamma matrix (document-topic probabilities)
td_gamma <- tidy(lda_model, matrix = "gamma")
td_gamma
# A tibble with columns: document, topic, gamma

# Classify documents by dominant topic
doc_topics <- td_gamma |
  group_by(document) |
  slice_max(gamma, n = 1) |
  ungroup()

# Augment original data with topic assignments
augmented <- augment(lda_model, data = AssociatedPress[1:100, ])
# Adds .topic column

# Get model summary
glance(lda_model)
# A tibble with: iter, terms, alpha
```

--------------------------------

### Compare Word Frequencies Across Datasets

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This snippet demonstrates how to join two different datasets to compare word frequencies. It normalizes the counts and uses log-scale scatter plots to visualize differences in word usage between two corpora.

```R
comparison <- tidy(AssociatedPress) |>
  count(word = term) |>
  rename(AP = n) |>
  inner_join(count(tidy_books, word)) |>
  rename(Austen = n) |>
  mutate(
    AP = AP / sum(AP),
    Austen = Austen / sum(Austen)
  )

library(scales)
ggplot(comparison, aes(AP, Austen)) +
  geom_point(alpha = 0.5) +
  geom_text(aes(label = word), check_overlap = TRUE, vjust = 1, hjust = 1) +
  scale_x_log10(labels = percent_format()) +
  scale_y_log10(labels = percent_format()) +
  geom_abline(color = "red")
```

--------------------------------

### Create Sparse Matrices from Tidy Data

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts tidy data frames into sparse matrix formats (dgCMatrix) suitable for machine learning algorithms. It maps document-term relationships into a matrix structure.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

book_word_matrix <- austen_books() |> unnest_tokens(word, text) |> count(book, word) |> cast_sparse(book, word, n)
```

--------------------------------

### get_stopwords - Access Stopword Lists

Source: https://context7.com/juliasilge/tidytext/llms.txt

Retrieves lists of common stopwords in multiple languages and sources.

```APIDOC
## get_stopwords

### Description
Retrieves stopword lists in multiple languages from the stopwords package.

### Parameters
- **language** (string) - Optional - ISO language code (default "en").
- **source** (string) - Optional - The source of the stopword list (e.g., "snowball", "smart").

### Response
- **word** (string) - The stopword.
- **lexicon** (string) - The source lexicon.
```

--------------------------------

### Create Document-Term Matrix (DTM) / Term-Document Matrix (TDM) with tm package

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts tidy data frames into DocumentTermMatrix or TermDocumentMatrix objects compatible with the tm package. Requires tidy data with columns for document, term, and value. Can optionally use weighting functions like weightTfIdf.

```r
library(tidytext)
library(dplyr)
library(tm)
library(janeaustenr)

# Create tidy word counts
book_words <- austen_books() |
  unnest_tokens(word, text) |
  count(book, word)

# Cast to DocumentTermMatrix
book_dtm <- cast_dtm(book_words, document = book, term = word, value = n)
book_dtm
# <<DocumentTermMatrix (documents: 6, terms: 14520)>>

# Cast to TermDocumentMatrix
book_tdm <- cast_tdm(book_words, term = word, document = book, value = n)
book_tdm
# <<TermDocumentMatrix (terms: 14520, documents: 6)>>

# Use with tm's weighting functions
book_dtm_tfidf <- cast_dtm(book_words, book, word, n, weighting = weightTfIdf)
```

--------------------------------

### get_sentiments - Access Sentiment Lexicons

Source: https://context7.com/juliasilge/tidytext/llms.txt

Retrieves various sentiment lexicons for sentiment analysis, including Bing, AFINN, NRC, and Loughran.

```APIDOC
## get_sentiments

### Description
Retrieves sentiment lexicons for sentiment analysis. Available lexicons include "bing", "afinn", "nrc", and "loughran".

### Parameters
- **lexicon** (string) - Required - The name of the lexicon to retrieve.

### Response
- **word** (string) - The word being analyzed.
- **sentiment/value** (mixed) - The sentiment classification or numeric score associated with the word.
```

--------------------------------

### Access Sentiment Lexicons and Perform Analysis

Source: https://context7.com/juliasilge/tidytext/llms.txt

Retrieves various sentiment lexicons like Bing, AFINN, or NRC to perform sentiment analysis. It demonstrates joining lexicons with tokenized text to calculate net sentiment scores.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

bing <- get_sentiments("bing")

austen_sentiment <- austen_books() |> unnest_tokens(word, text) |> inner_join(get_sentiments("bing"), by = "word") |> count(book, sentiment) |> tidyr::pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |> mutate(net_sentiment = positive - negative)
```

--------------------------------

### Visualize Sentiment Trajectories with ggplot2

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This snippet demonstrates how to visualize sentiment scores across different books using ggplot2. It uses facet_wrap to create separate plots for each book, providing a clear view of sentiment trends.

```R
library(ggplot2)

ggplot(janeaustensentiment, aes(index, sentiment, fill = book)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(vars(book), ncol = 2, scales = "free_x")
```

--------------------------------

### Dataset: parts_of_speech

Source: https://context7.com/juliasilge/tidytext/llms.txt

A dataset containing parts of speech for English words from the Moby project, useful for linguistic filtering.

```APIDOC
## [DATASET] parts_of_speech

### Description
A comprehensive list of English words mapped to their respective parts of speech (e.g., Noun, Verb, Adjective).

### Structure
- **word** (character) - The word.
- **pos** (character) - The part of speech category.

### Usage Example
```r
# Join with tokenized text to analyze POS distribution
austen_books() |>
  unnest_tokens(word, text) |>
  inner_join(parts_of_speech, by = "word") |>
  count(pos, sort = TRUE)
```
```

--------------------------------

### Tidy tm and quanteda Corpus Objects

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts Corpus objects from the tm or quanteda packages into tidy data frames. The output tibble contains the text content and any associated metadata columns, making it easier to process corpus data.

```r
library(tidytext)
library(dplyr)
library(tm)

# tm package Corpus
txt <- system.file("texts", "txt", package = "tm")
ovid <- VCorpus(DirSource(txt, encoding = "UTF-8"),
                readerControl = list(language = "lat"))

# Tidy the corpus
tidy(ovid)
# A tibble with columns: text (plus metadata columns)
```

--------------------------------

### Tidy Document-Term Matrix (DTM) / Term-Document Matrix (TDM)

Source: https://context7.com/juliasilge/tidytext/llms.txt

Converts DocumentTermMatrix, TermDocumentMatrix, and dfm objects from the tm and quanteda packages back into a tidy data frame format. The resulting tibble contains columns for document, term, and count, enabling further analysis.

```r
library(tidytext)
library(dplyr)
library(tm)

# Load example DTM
data("AssociatedPress", package = "topicmodels")
AssociatedPress
# <<DocumentTermMatrix (documents: 2246, terms: 10473)>>

# Tidy the DTM
ap_tidy <- tidy(AssociatedPress)
ap_tidy
# A tibble with columns: document, term, count

# Analyze the tidy data
ap_tidy |
  group_by(term) |
  summarise(total = sum(count)) |
  arrange(desc(total))

# Sentiment analysis on tidied DTM
ap_sentiments <- ap_tidy |
  inner_join(get_sentiments("bing"), by = c(term = "word")) |
  count(document, sentiment, wt = count) |
  tidyr::pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |
  mutate(sentiment = positive - negative)
```

--------------------------------

### Perform Sentiment Analysis with Bing Lexicon (R)

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Performs sentiment analysis on the text data using the Bing lexicon. It joins the tokenized words with sentiment scores, counts positive and negative sentiments per section of each book, and calculates a sentiment score.

```r
library(tidyr)
janeaustensentiment <- tidy_books |
  inner_join(
    get_sentiments("bing"),
    by = "word",
    relationship = "many-to-many"
  ) |
  count(book, index = line %/% 80, sentiment) |
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |
  mutate(sentiment = positive - negative)
```

--------------------------------

### Tokenize Text with unnest_tokens

Source: https://context7.com/juliasilge/tidytext/llms.txt

The primary function for converting text into tidy format. It supports various tokenization methods including words, sentences, n-grams, regex, and HTML parsing.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)

books <- tibble(text = prideprejudice, book = "Pride & Prejudice") |>
  mutate(line = row_number())

tidy_words <- books |>
  unnest_tokens(output = word, input = text)

tidy_sentences <- books |>
  unnest_tokens(output = sentence, input = text, token = "sentences")

tidy_bigrams <- books |>
  unnest_tokens(output = bigram, input = text, token = "ngrams", n = 2)

tidy_chapters <- books |>
  unnest_tokens(output = chapter, input = text, token = "regex", pattern = "Chapter [\\d]")

html_data <- tibble(row = 1:2, text = c("<h1>Text <b>is</b>", "<a href='example.com'>here</a>"))
html_data |>
  unnest_tokens(word, text, format = "html")

books |>
  unnest_tokens(word, text, token = stringr::str_split, pattern = " ")

books |>
  unnest_tokens(word, text, to_lower = FALSE)
```

--------------------------------

### Remove Stop Words using anti_join (R)

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Removes common English stop words from the tokenized text data. It uses an anti_join with the stop words obtained from get_stopwords() function.

```r
tidy_books <- tidy_books |
  anti_join(get_stopwords())
```

--------------------------------

### bind_tf_idf - Calculate TF-IDF Scores

Source: https://context7.com/juliasilge/tidytext/llms.txt

Calculates term frequency (tf), inverse document frequency (idf), and their product (tf-idf) to identify important words in a document collection.

```APIDOC
## bind_tf_idf

### Description
Calculates term frequency (tf), inverse document frequency (idf), and their product (tf-idf) for identifying words that are important to documents within a collection.

### Parameters
- **term** (column) - Required - The column containing the terms (words).
- **document** (column) - Required - The column containing the document identifiers.
- **n** (column) - Required - The column containing the counts of terms.

### Response
- **tf** (numeric) - Term frequency.
- **idf** (numeric) - Inverse document frequency.
- **tf_idf** (numeric) - The product of tf and idf.
```

--------------------------------

### Dataset: nma_words

Source: https://context7.com/juliasilge/tidytext/llms.txt

A dataset containing English negators, modals, and adverbs used to provide context for sentiment analysis.

```APIDOC
## [DATASET] nma_words

### Description
Provides a tibble of 44 words classified by modifier type (negator, modal, adverb) to help identify sentiment reversals or intensifiers in text.

### Structure
- **word** (character) - The word itself.
- **modifier** (character) - The type of modifier (e.g., "negator").

### Usage Example
```r
# Filter for negators to reverse sentiment
negated_words <- bigrams |> 
  filter(word1 %in% nma_words$word[nma_words$modifier == "negator"]) |>
  inner_join(get_sentiments("bing"), by = c(word2 = "word"))
```
```

--------------------------------

### Tokenize Text Data using unnest_tokens (R)

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Converts the text data from a one-row-per-line format to a one-token-per-row format using the unnest_tokens function from the tidytext package. This function tokenizes text into words by default.

```r
library(tidytext)
tidy_books <- original_books |
  unnest_tokens(word, text)
```

--------------------------------

### Sentiment Analysis with nma_words Dataset

Source: https://context7.com/juliasilge/tidytext/llms.txt

Utilizes the nma_words dataset to identify negators, modals, and adverbs. This is useful for identifying context-dependent sentiment shifts in bigram analysis.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

# Find negated sentiment words
bigrams <- austen_books() |>
  unnest_tokens(bigram, text, token = "ngrams", n = 2) |>
  tidyr::separate(bigram, c("word1", "word2"), sep = " ")

negated_words <- bigrams |>
  filter(word1 %in% nma_words$word[nma_words$modifier == "negator"]) |>
  inner_join(get_sentiments("bing"), by = c(word2 = "word"))
```

--------------------------------

### get_sentiments

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Retrieves a sentiment lexicon for use in text analysis.

```APIDOC
## GET /get_sentiments

### Description
Retrieves a predefined sentiment lexicon (e.g., 'bing', 'afinn', 'nrc') to map words to sentiment scores or categories.

### Method
GET

### Endpoint
/get_sentiments

### Parameters
#### Query Parameters
- **lexicon** (string) - Required - The name of the lexicon to retrieve.

### Request Example
GET /get_sentiments?lexicon=bing

### Response
#### Success Response (200)
- **word** (string) - The token being evaluated.
- **sentiment** (string) - The sentiment category (e.g., positive, negative).

#### Response Example
{
  "word": "abnormal",
  "sentiment": "negative"
}
```

--------------------------------

### Manage and Filter Stopwords

Source: https://context7.com/juliasilge/tidytext/llms.txt

Retrieves stopword lists in multiple languages and sources, or uses the built-in stop_words dataset to filter out common words from text analysis.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

get_stopwords(language = "en", source = "snowball")

austen_books() |> unnest_tokens(word, text) |> anti_join(stop_words, by = "word") |> count(word, sort = TRUE)
```

--------------------------------

### Perform Character-Level Tokenization

Source: https://context7.com/juliasilge/tidytext/llms.txt

Functions for splitting text into individual characters or character shingles, useful for stylometry and language identification.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)

d <- tibble(txt = prideprejudice)

d |>
  unnest_characters(char, txt)

d |>
  unnest_character_shingles(shingle, txt, n = 3)

d |>
  unnest_characters(char, txt, strip_non_alphanum = FALSE)
```

--------------------------------

### Perform Sentiment Analysis on DocumentTermMatrix

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This snippet illustrates how to calculate sentiment scores for documents within a DocumentTermMatrix. It joins the tidy data with sentiment lexicons, aggregates counts, and pivots the data to calculate net sentiment scores.

```R
ap_sentiments <- tidy(AssociatedPress) |>
  inner_join(get_sentiments("bing"), by = c(term = "word")) |>
  count(document, sentiment, wt = count) |>
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) |>
  mutate(sentiment = positive - negative) |>
  arrange(sentiment)
```

--------------------------------

### Convert DocumentTermMatrix to Tidy Format

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

This snippet shows how to convert a DocumentTermMatrix object from the tm package into a tidy data frame using the tidy() function. This transformation is essential for applying tidyverse functions to legacy text mining data.

```R
library(tm)
data("AssociatedPress", package = "topicmodels")
tidy(AssociatedPress)
```

--------------------------------

### Calculate TF-IDF Scores with tidytext

Source: https://context7.com/juliasilge/tidytext/llms.txt

Calculates term frequency, inverse document frequency, and the resulting tf-idf score to identify important words within a document collection. It requires a tidy data frame containing term, document, and count columns.

```R
library(tidytext)
library(dplyr)
library(janeaustenr)

book_words <- austen_books() |> unnest_tokens(word, text) |> count(book, word, sort = TRUE)

book_tf_idf <- book_words |> bind_tf_idf(term = word, document = book, n = n)

book_tf_idf |> arrange(desc(tf_idf)) |> group_by(book) |> slice_max(tf_idf, n = 10)
```

--------------------------------

### Extract N-grams and Skip-grams

Source: https://context7.com/juliasilge/tidytext/llms.txt

Functions for extracting contiguous sequences of items (n-grams) or sequences with gaps (skip-grams) from text data.

```r
library(tidytext)
library(dplyr)
library(janeaustenr)

d <- tibble(txt = prideprejudice)

d |>
  unnest_ngrams(word, txt, n = 2)

d |>
  unnest_ngrams(word, txt, n = 3)

d |>
  unnest_skip_ngrams(word, txt, n = 3, k = 1)

d |>
  unnest_ngrams(word, txt, n = 2, ngram_delim = "_")
```

--------------------------------

### Count Word Frequencies (R)

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Counts the occurrences of each word in the processed text data and sorts them in descending order. This helps identify the most frequent words.

```r
tidy_books |
  count(word, sort = TRUE)
```

--------------------------------

### unnest_tokens

Source: https://github.com/juliasilge/tidytext/blob/main/README.md

Converts a dataframe with a text column into a one-token-per-row format.

```APIDOC
## POST /unnest_tokens

### Description
Splits a column of text into individual tokens (words, sentences, n-grams) to create a tidy dataset.

### Method
POST

### Endpoint
/unnest_tokens

### Parameters
#### Request Body
- **output** (string) - Required - The name of the new column to create.
- **input** (string) - Required - The name of the column containing the text to be tokenized.
- **token** (string) - Optional - The unit of tokenization (e.g., 'words', 'sentences', 'ngrams').

### Request Example
{
  "output": "word",
  "input": "text",
  "token": "words"
}

### Response
#### Success Response (200)
- **data** (dataframe) - A tibble with the original columns plus the new token column.

#### Response Example
{
  "book": "Sense & Sensibility",
  "line": 1,
  "word": "sense"
}
```

--------------------------------

### Reorder Categorical Variables within ggplot2 Facets

Source: https://context7.com/juliasilge/tidytext/llms.txt

Demonstrates the use of reorder_within and scale_x_reordered/scale_y_reordered to correctly sort categorical data within faceted ggplot2 plots. This solves the common issue where reorder() fails to respect facet boundaries.

```R
library(tidytext)
library(dplyr)
library(tidyr)
library(ggplot2)
library(janeaustenr)

book_words <- austen_books() |>
  unnest_tokens(word, text) |>
  anti_join(stop_words) |>
  count(book, word, sort = TRUE) |>
  group_by(book) |>
  slice_max(n, n = 10) |>
  ungroup()

# With reorder_within: correct order within each facet
ggplot(book_words, aes(n, reorder_within(word, n, book))) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ book, scales = "free")

# Multiple faceting variables
ggplot(mtcars, aes(mpg, reorder_within(carb, mpg, list(vs, am)))) +
  geom_boxplot() +
  scale_x_reordered() +
  facet_wrap(vs ~ am, scales = "free_x")
```

--------------------------------

### Faceted Bar Plot Helpers: reorder_within / scale_y_reordered

Source: https://context7.com/juliasilge/tidytext/llms.txt

Helper functions to correctly order categorical variables within ggplot2 facets, solving the common issue where reorder() does not respect facet groups.

```APIDOC
## [FUNCTION] reorder_within / scale_y_reordered

### Description
These functions allow for the reordering of categorical variables within each facet of a ggplot2 plot, ensuring that bars are sorted correctly regardless of the global order.

### Parameters
#### Arguments
- **x** (vector) - Required - The categorical variable to reorder.
- **by** (vector) - Required - The numeric variable to order by.
- **within** (vector/list) - Required - The variable(s) defining the facets.

### Usage Example
```r
ggplot(data, aes(n, reorder_within(word, n, book))) +
  geom_col() +
  scale_y_reordered() +
  facet_wrap(~ book, scales = "free")
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.