readtext

https://readtext.quanteda.io
Admin
readtext is an R package designed to import and handle plain and formatted text files along with...
Tokens:19,522
Snippets:154
Trust Score:9
Update:4 days ago
Show doc for...
Tokens:
Raw
### Install readtext development version from GitHub

Source: https://readtext.quanteda.io/index.html

Install the latest development version from GitHub using remotes::install_github(). Requires the remotes package.

```R
# devtools packaged required to install readtext from Github 
remotes::install_github("quanteda/readtext")
```

--------------------------------

### get_nexis_html usage examples

Source: https://readtext.quanteda.io/reference/get_nexis_html.html

Examples showing how to use get_nexis_html to parse specific HTML files and how to use readtext with the nexis source.

```R
if (FALSE) { # \dontrun{
irt <- readtext:::get_nexis_html('tests/data/nexis/irish-times_1995-06-12_0001.html')
afp <- readtext:::get_nexis_html('tests/data/nexis/afp_2013-03-12_0501.html')
gur <- readtext:::get_nexis_html('tests/data/nexis/guardian_1986-01-01_0001.html')
sun <- readtext:::get_nexis_html('tests/data/nexis/sun_2000-11-01_0001.html')
spg <- readtext:::get_nexis_html('tests/data/nexis/spiegel_2012-02-01_0001.html', 
                                  language_date = 'german')

all <- readtext('tests/data/nexis', source = 'nexis')
all <- readtext('tests/data/nexis', source = 'nexis')
} # }
```

--------------------------------

### Install readtext from CRAN

Source: https://readtext.quanteda.io/index.html

Install the stable version of readtext from CRAN using install.packages().

```R
install.packages("readtext")
```

--------------------------------

### Install Linux dependencies for readtext

Source: https://readtext.quanteda.io/index.html

On Debian/Ubuntu, install libpoppler-cpp-dev to support reading PDF files (used by antiword).

```bash
sudo apt-get install libpoppler-cpp-dev   # for antiword
```

--------------------------------

### Usage examples for basename_unique

Source: https://readtext.quanteda.io/reference/basename_unique.html

Examples demonstrating how to extract unique path components from file paths using basename_unique with and without the path_only argument.

```R
files <- c("../data/glob/subdir1/test.txt", "../data/glob/subdir2/test.txt")
readtext:::basename_unique(files)
#> [1] "subdir1" "subdir2"
# [1] "subdir1/test.txt" "subdir2/test.txt"
readtext:::basename_unique(files, path_only = TRUE)
#> [1] "subdir1" "subdir2"
# [1] "subdir1" "subdir2"
readtext:::basename_unique(c("../data/test1.txt", "../data/test2.txt"))
#> [1] "test1.txt" "test2.txt"
# [1] "test1.txt" "test2.txt"
```

--------------------------------

### Examples of managing readtext options

Source: https://readtext.quanteda.io/reference/readtext_options.html

Demonstrates saving current options, modifying verbosity, and resetting to previous configurations.

```R
if (FALSE) { # \dontrun{
# save the current options
(opt <- readtext_options())

# set higher verbosity
readtext_options(verbosity = 3)

# read something in here
if (!interactive()) pkgload::load_all()
DATA_DIR <- system.file("extdata/", package = "readtext")
readtext(paste0(DATA_DIR, "/txt/UDHR/*"))

# reset to saved options
readtext_options(opt)
} # }
```

--------------------------------

### Get the data directory from readtext

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Locates the package's internal extdata directory containing example files for testing.

```R
DATA_DIR <- system.file("extdata/", package = "readtext")
```

--------------------------------

### Read texts with encodings and create a corpus

Source: https://readtext.quanteda.io/reference/data_files_encodedtexts.html

Reads the .txt files again with the extracted encodings, adds document variables from the filenames (document, language, inputEncoding), and creates a quanteda corpus from the resulting readtext object. The example is wrapped in a \dontrun{} block.

```r
txts <- readtext(paste0(FILEDIR, "/", "*.txt"), encoding = fileencodings,
                  docvarsfrom = "filenames", 
                  docvarnames = c("document", "language", "inputEncoding"))
#> Error: object 'FILEDIR' not found
encodingCorpus <- corpus(txts, source = "Created by encoding-tests.R") 
#> Error: object 'txts' not found
summary(encodingCorpus)
#> Error: object 'encodingCorpus' not found
 # \dontrun{}
```

--------------------------------

### readtext_options()

Source: https://readtext.quanteda.io/reference/index.html

Get or set package options for readtext, such as caching and progress display.

```APIDOC
## readtext_options()

### Description
Get or set package options for readtext.

### Usage
readtext_options(...)

### Arguments
- **...** (optional) - Options to set. If none provided, returns current options.

### Value
A list of current options if no arguments are given; otherwise, invisibly returns the updated options.

### Examples
```r
# Get current options
readtext_options()

# Set an option
readtext_options(cache = FALSE)
```
```

--------------------------------

### Read text data using readtext

Source: https://readtext.quanteda.io/reference/readtext.html

Examples demonstrating how to read various file formats including text, CSV, TSV, JSON, PDF, and Word documents, as well as how to extract document variables from file paths and names.

```R
if (FALSE) { # \dontrun{
## get the data directory
if (!interactive()) pkgload::load_all()
DATA_DIR <- system.file("extdata/", package = "readtext")

## read in some text data
# all UDHR files
(rt1 <- readtext(paste0(DATA_DIR, "/txt/UDHR/*")))

# manifestos with docvars from filenames
(rt2 <- readtext(paste0(DATA_DIR, "/txt/EU_manifestos/*.txt"),
                 docvarsfrom = "filenames", 
                 docvarnames = c("unit", "context", "year", "language", "party"),
                 encoding = "LATIN1"))
                 
# recurse through subdirectories
(rt3 <- readtext(paste0(DATA_DIR, "/txt/movie_reviews/*"), 
                 docvarsfrom = "filepaths", docvarnames = "sentiment"))

## read in csv data
(rt4 <- readtext(paste0(DATA_DIR, "/csv/inaugCorpus.csv")))

## read in tab-separated data
(rt5 <- readtext(paste0(DATA_DIR, "/tsv/dailsample.tsv"), text_field = "speech"))

## read in JSON data
(rt6 <- readtext(paste0(DATA_DIR, "/json/inaugural_sample.json"), text_field = "texts"))

## read in pdf data
# UNHDR
(rt7 <- readtext(paste0(DATA_DIR, "/pdf/UDHR/*.pdf"), 
                 docvarsfrom = "filenames", 
                 docvarnames = c("document", "language")))
Encoding(rt7$text)

## read in Word data (.doc)
(rt8 <- readtext(paste0(DATA_DIR, "/word/*.doc")))
Encoding(rt8$text)

## read in Word data (.docx)
(rt9 <- readtext(paste0(DATA_DIR, "/word/*.docx")))
Encoding(rt9$text)

## use elements of path and filename as docvars
(rt10 <- readtext(paste0(DATA_DIR, "/pdf/UDHR/*.pdf"), 
                  docvarsfrom = "filepaths", dvsep = "[/_.]"))
} # }
```

--------------------------------

### Importing Manifestos with docvars from Filenames

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

This example demonstrates reading text files while extracting metadata from filenames. The docvarsfrom argument is set to 'filenames', and dvsep defines the separator used to parse the docvarnames.

```R
# Manifestos with docvars from filenames
readtext(paste0(DATA_DIR, "/txt/EU_manifestos/*.txt"),
         docvarsfrom = "filenames", 
         docvarnames = c("unit", "context", "year", "language", "party"),
         dvsep = "_", 
         encoding = "ISO-8859-1")
## readtext object consisting of 17 documents and 5 docvars.
## $text
##  [1] "\033[38;5;246m# A data frame: 17 × 7\033[39m"                                                                                                                                                                                                                                                                                                       
##  [2] "  doc_id                  text                unit  context  year language party"                                                                                                                                                                                                                                                                   
##  [3] "  \033[3m\033[38;5;246m<chr>\033[39m\033[23m                   \033[3m\033[38;5;246m<chr>\033[39m\033[23m               \033[3m\033[38;5;246m<chr>\033[39m\033[23m \033[3m\033[38;5;246m<chr>\033[39m\033[23m   \033[3m\033[38;5;246m<int>\033[39m\033[23m \033[3m\033[38;5;246m<chr>\033[39m\033[23m    \033[3m\033[38;5;246m<chr>\033[39m\033[23m"
##  [4] "\033[38;5;250m1\033[39m EU_euro_2004_de_PSE.txt \033[38;5;246m\"\033[39m\\\"PES · PSE \\\"...\033[38;5;246m\"\033[39m EU    euro     \033[4m2\033[24m004 de       PSE  "                                                                                                                                                                            
##  [5] "\033[38;5;250m2\033[39m EU_euro_2004_de_V.txt   \033[38;5;246m\"\033[39m\\\"Gemeinsame\\\"...\033[38;5;246m\"\033[39m EU    euro     \033[4m2\033[24m004 de       V    "                                                                                                                                                                            
##  [6] "\033[38;5;250m3\033[39m EU_euro_2004_en_PSE.txt \033[38;5;246m\"\033[39m\\\"PES · PSE \\\"...\033[38;5;246m\"\033[39m EU    euro     \033[4m2\033[24m004 en       PSE  "                                                                                                                                                                            
##  [7] "\033[38;5;250m4\033[39m EU_euro_2004_en_V.txt   \033[38;5;246m\"\033[39m\\\"Manifesto\\n\\\"..… EU    euro     \033[4m2\033[24m004 en       V    "                                                                                                                                                                                                  
##  [8] "\033[38;5;250m5\033[39m EU_euro_2004_es_PSE.txt \033[38;5;246m\"\033[39m\\\"PES · PSE \\\"...\033[38;5;246m\"\033[39m EU    euro     \033[4m2\033[24m004 es       PSE  "                                                                                                                                                                            
##  [9] "\033[38;5;250m6\033[39m EU_euro_2004_es_V.txt   \033[38;5;246m\"\033[39m\\\"Manifesto\\n\\\"..… EU    euro     \033[4m2\033[24m004 es       V    "                                                                                                                                                                                                  
## [10] "\033[38;5;246m# ℹ 11 more rows\033[39m"                                                                                                                                                                                                                                                                                                             
## 
## $summary
## $summary[[1]]
## NULL
## 
## 
## attr(,"class")
## [1] "trunc_mat"
```

--------------------------------

### Read encoded texts with readtext and inspect output

Source: https://readtext.quanteda.io/reference/data_files_encodedtexts.html

Loads the quanteda package, reads all .txt files from the temporary directory using readtext without specifying encodings, and prints the first 80 characters of selected documents to show gibberish or hex output. The example then reads the files again with the extracted encodings and shows that the texts are correctly decoded for English, Arabic, Cyrillic, Chinese, and Hindi.

```r
# try readtext
require(quanteda)
#> Loading required package: quanteda
#> Package version: 4.3.1
#> Unicode version: 14.0
#> ICU version: 71.1
#> Parallel computing: 10 of 10 threads used.
#> See https://quanteda.io for tutorials and examples.
#> 
#> Attaching package: ‘quanteda’
#> The following object is masked from ‘package:readtext’:
#> 
#>     texts
txts <- readtext(paste0(FILEDIR, "/", "*.txt"))
#> Error: object 'FILEDIR' not found
substring(texts(txts)[1], 1, 80) # gibberish
#> Error: object 'txts' not found
substring(texts(txts)[4], 1, 80) # hex
#> Error: object 'txts' not found
substring(texts(txts)[40], 1, 80) # hex
#> Error: object 'txts' not found

# read them in again
txts <- readtext(paste0(FILEDIR,  "/", "*.txt"), encoding = fileencodings)
#> Error: object 'FILEDIR' not found
substring(texts(txts)[1], 1, 80)  # English
#> Error: object 'txts' not found
substring(texts(txts)[4], 1, 80)  # Arabic, looking good 
#> Error: object 'txts' not found
substring(texts(txts)[40], 1, 80) # Cyrillic, looking good
#> Error: object 'txts' not found
substring(texts(txts)[7], 1, 80)  # Chinese, looking good
#> Error: object 'txts' not found
substring(texts(txts)[26], 1, 80) # Hindi, looking good
#> Error: object 'txts' not found
```

--------------------------------

### Usage of readtext_options function

Source: https://readtext.quanteda.io/reference/readtext_options.html

The basic syntax for calling readtext_options to get or set package configurations.

```R
readtext_options(..., reset = FALSE, initialize = FALSE)
```

--------------------------------

### Read all text files from a directory

Source: https://readtext.quanteda.io/index.html

Loads all files matching a glob pattern from the readtext package's example data directory. The output shows a readtext object with 13 documents and 0 docvars.

```R
library("readtext")
# get the data directory from readtext
DATA_DIR <- system.file("extdata/", package = "readtext")

# read in all files from a folder
readtext(paste0(DATA_DIR, "/txt/UDHR/*"))
```

```R
## readtext object consisting of 13 documents and 0 docvars.
## # A data frame: 13 × 2
##   doc_id            text                         
##   <chr>             <chr>                        
## 1 UDHR_chinese.txt  "\"世界人权宣言\n联合国\"..."
## 2 UDHR_czech.txt    "\"VŠEOBECNÁ \"..."          
## 3 UDHR_danish.txt   "\"Den 10. de\"..."          
## 4 UDHR_english.txt  "\"Universal \"..."          
## 5 UDHR_french.txt   "\"Déclaratio\"..."          
## 6 UDHR_georgian.txt "\"FLFVBFYBC \"..."          
## # ℹ 7 more rows
```

--------------------------------

### Unzip encoded texts and extract encodings from filenames

Source: https://readtext.quanteda.io/reference/data_files_encodedtexts.html

Unzips the example .zip file to a temporary directory and extracts the encoding from each filename by stripping the extension and splitting on underscores. The code is wrapped in an if(FALSE) block, so it is not executed by default.

```r
if (FALSE) # unzip the files to a temporary directory
FILEDIR <- tempdir()
unzip(system.file("extdata", "data_files_encodedtexts.zip", package = "readtext"), 
      exdir = FILEDIR)
#> Error: object 'FILEDIR' not found

# get encoding from filename
filenames <- list.files(FILEDIR, "\\.txt$")
#> Error: object 'FILEDIR' not found
# strip the extension
filenames <- gsub(".txt$", "", filenames)
#> Error: object 'filenames' not found
parts <- strsplit(filenames, "_")
#> Error: object 'filenames' not found
fileencodings <- sapply(parts, "[", 3)
#> Error: object 'parts' not found
fileencodings
#> Error: object 'fileencodings' not found
```

--------------------------------

### Read PDF files with readtext

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Loads all PDF files from a folder, extracting document-level variables from filenames using a delimiter. The example is incomplete in the source.

```R
readtext(paste0(DATA_DIR, "/pdf/UDHR/*.pdf"),
         docvarsfrom = "filenames", 
         docvarnames = c("document", "language"),
         dvsep = "_")
```

--------------------------------

### readtext_options(...)

Source: https://readtext.quanteda.io/reference/readtext_options.html

Gets or sets global options for the readtext package. When called with key-value pairs, sets options and returns TRUE invisibly. When called with no arguments, returns a named list of current options. When called with reset=TRUE, resets all options to defaults and returns TRUE invisibly.

```APIDOC
## readtext_options(...)

### Description
Gets or sets global options affecting functions across the readtext package.

### Usage
```r
readtext_options(..., reset = FALSE, initialize = FALSE)
```

### Arguments
- **...** - Options to be set, as key-value pairs, same as `options()`. This may be a list of valid key-value pairs, useful for setting a group of options at once.
- **reset** (logical) - If `TRUE`, reset all readtext options to their default values.
- **initialize** (logical) - If `TRUE`, reset only the readtext options that are not already defined. Used for setting initial values when some have been defined previously, such as in `.Rprofile`.

### Value
- When called using a `key = value` pair (where `key` can be a label or quoted character name), the option is set and `TRUE` is returned invisibly.
- When called with no arguments, a named list of the package options is returned.
- When called with `reset = TRUE` as an argument, all options are reset to their default values, and `TRUE` is returned invisibly.

### Details
Currently available options are:
- **verbosity** - Default verbosity for messages produced when reading files. See `readtext()`.

### Examples
```r
if (FALSE) { # \dontrun{
# save the current options
(opt <- readtext_options())

# set higher verbosity
readtext_options(verbosity = 3)

# read something in here
if (!interactive()) pkgload::load_all()
DATA_DIR <- system.file("extdata/", package = "readtext")
readtext(paste0(DATA_DIR, "/txt/UDHR/*"))

# reset to saved options
readtext_options(opt)
} # }
```
```

--------------------------------

### Create a quanteda corpus

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Initializes a quanteda corpus from the readtext object 'txts' and displays a summary of the first 5 documents.

```R
if (require("quanteda")) {
corpus_txts <- corpus(txts)
summary(corpus_txts, 5)
}
```

--------------------------------

### texts(x, ...)

Source: https://readtext.quanteda.io/reference/texts.html

Deprecated function to get the texts from a readtext object. Use as.character.readtext() instead.

```APIDOC
## texts(x, ...)

### Description
Get the texts from a readtext object. This function is **deprecated**.

### Method
Function (S3 method for class 'readtext')

### Signature
```r
texts(x, ...)

# S3 method for class 'readtext'
texts(x, ...)
```

### Parameters
- **x** (readtext object) - Required - a readtext object
- **...** - Optional - not used

### Value
A character vector of the texts in the corpus.

### Details
This function is **deprecated**. Use `as.character.readtext()` to turn a readtext object into a simple named character vector of documents.
```

--------------------------------

### Load readtext package

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Initializes the readtext package for use in the R environment.

```R
library("readtext")
```

--------------------------------

### Create quanteda corpus from readtext object

Source: https://readtext.quanteda.io/index.html

Loads quanteda, reads a CSV with readtext, creates a corpus with corpus(), and prints a summary. The output shows the corpus structure with document variables.

```r
library("quanteda")
```

```r
# read in comma-separated values with readtext
rt_csv <- readtext(paste0(DATA_DIR, "/csv/inaugCorpus.csv"), text_field = "texts")
# create quanteda corpus
corpus_csv <- corpus(rt_csv)
summary(corpus_csv, 5)
```

--------------------------------

### Load stringi package for text cleaning

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Loads the stringi package, which is recommended for performing regular expression operations on imported text.

```R
# Load stringi package
require("stringi")
```

--------------------------------

### readtext-package

Source: https://readtext.quanteda.io/reference/index.html

Overview of the readtext package: Import and handling for plain and formatted text files.

```APIDOC
## readtext-package

### Description
Import and handling for plain and formatted text files.

### Details
This package provides functions to read text files into R, including support for various formats and encodings.

### See Also
Useful links:
- Package homepage: https://readtext.quanteda.io
- Bug reports: https://github.com/quanteda/readtext/issues
```

--------------------------------

### print(x, n = 6L, text_width = 10L, ...)

Source: https://readtext.quanteda.io/reference/print.readtext.html

Prints a readtext object in a nicely formatted way, showing a specified number of rows and truncating the text field to a given character width.

```APIDOC
## print(x, n = 6L, text_width = 10L, ...)

### Description
Prints a readtext object in a nicely formatted way.

### Usage
```r
# S3 method for class 'readtext'
print(x, n = 6L, text_width = 10L, ...)
```

### Arguments
- **x** - The readtext object to be printed.
- **n** - A single integer, the number of rows of a readtext object to print. Default is 6L.
- **text_width** - Number of characters to display of the text field. Default is 10L.
- **...** - Not used here.

```

--------------------------------

### get_temp(prefix, temp_dir, directory, seed)

Source: https://readtext.quanteda.io/reference/get_temp.html

Returns a path to a temporary file or directory. The function appends a random string to the given prefix, optionally creates a directory, and uses a seed for reproducible generation.

```APIDOC
## get_temp(prefix = "readtext-", temp_dir = NULL, directory = FALSE, seed = NULL)

### Description
Returns a path to a temporary file or directory. The function generates a unique path by appending a random string to the given prefix, optionally creating a directory, and using a seed for reproducible generation.

### Usage
```r
get_temp(prefix = "readtext-", temp_dir = NULL, directory = FALSE, seed = NULL)
```

### Arguments
- **prefix** (character) - Optional - A string appended to random file or directory names.
- **temp_dir** (character) - Optional - A path to temporary directory. If `NULL`, value from `tempdir()` will be used.
- **directory** (logical) - Optional - If `TRUE`, temporary directory will be created.
- **seed** (numeric) - Optional - A seed value for `digest::digest`. If `NULL`, a random value will be used.

### Return Value
A character string representing the path to the temporary file or directory.

### Example
```r
# Get a temporary file path
get_temp()

# Get a temporary directory path and create it
get_temp(directory = TRUE)
```
```

--------------------------------

### Extract encoded text files from zip

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Sets up a temporary directory and extracts the contents of the encoded texts zip file provided by the readtext package.

```R
FILEDIR <- tempdir()
# unzip file
unzip(system.file("extdata", "data_files_encodedtexts.zip", package = "readtext"), exdir = FILEDIR)
```

--------------------------------

### Read all files from a folder with readtext()

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Reads all files in the specified folder (using a wildcard) into a readtext object. The output shows a data frame with 13 documents and 0 docvars, listing document IDs and text previews.

```r
readtext(paste0(DATA_DIR, "/txt/UDHR/*"))
```

--------------------------------

### Read PDF files with readtext and extract docvars from filenames

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Reads all PDF files matching the pattern in DATA_DIR, using the filenames to create docvars 'document' and 'language' by splitting on underscores. The output is a readtext object with 11 documents and 2 docvars.

```R
(rt_pdf <- readtext(paste0(DATA_DIR, "/pdf/UDHR/*.pdf"), 
                    docvarsfrom = "filenames", 
                    docvarnames = c("document", "language"),
                    sep = "_"))
```

--------------------------------

### encoding()

Source: https://readtext.quanteda.io/reference/index.html

Detect the encoding of text files.

```APIDOC
## encoding()

### Description
Detect the encoding of texts.

### Usage
encoding(x, ...)

### Arguments
- **x** (character) - Required - A character vector or path to files.
- **...** (optional) - Additional arguments passed to underlying detection functions.

### Value
A character vector of detected encodings.

### Examples
```r
# Detect encoding of a file
encoding("path/to/file.txt")
```
```

--------------------------------

### Recurse through subdirectories with readtext

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Reads all .txt files in the movie_reviews subdirectory using a glob pattern. The output shows a readtext object with 10 documents and no docvars.

```r
readtext(paste0(DATA_DIR, "/txt/movie_reviews/*"))
```

--------------------------------

### get_nexis_html(path, paragraph_separator = "\n\n", verbosity, ...)

Source: https://readtext.quanteda.io/reference/get_nexis_html.html

Extracts headings, body texts, and meta data from Nexis HTML files. Accepts a file path or directory, and returns a data frame with the extracted content.

```APIDOC
## get_nexis_html(path, paragraph_separator = "\n\n", verbosity, ...)

### Description
Extracts headings, body texts, and meta data (date, byline, length, section, edition) from items in HTML files downloaded by the Nexis scraper.

### Usage
```r
get_nexis_html(path, paragraph_separator = "\n\n", verbosity, ...)
```

### Arguments
- **path** (character) - Required - Path to a HTML file or a directory that contains HTML files.
- **paragraph_separator** (character) - Optional - A character to separate paragraphs in body texts. Default is `"\n\n"`.
- **verbosity** (integer) - Optional - Level of output messages:
  - 0: output errors only
  - 1: output errors and warnings (default)
  - 2: output a brief summary message
  - 3: output detailed file-related messages
- **...** - Optional - Only to trap extra arguments.

### Examples
```r
if (FALSE) { # \dontrun{
irt <- readtext:::get_nexis_html('tests/data/nexis/irish-times_1995-06-12_0001.html')
afp <- readtext:::get_nexis_html('tests/data/nexis/afp_2013-03-12_0501.html')
gur <- readtext:::get_nexis_html('tests/data/nexis/guardian_1986-01-01_0001.html')
sun <- readtext:::get_nexis_html('tests/data/nexis/sun_2000-11-01_0001.html')
spg <- readtext:::get_nexis_html('tests/data/nexis/spiegel_2012-02-01_0001.html', 
                                  language_date = 'german')

all <- readtext('tests/data/nexis', source = 'nexis')
all <- readtext('tests/data/nexis', source = 'nexis')
} # }
```
```

--------------------------------

### Read Word data (.docx) with readtext

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Reads all .docx files from a directory using readtext. The output shows a readtext object with 2 documents and 0 docvars, displaying a data frame with doc_id and text columns.

```r
readtext(paste0(DATA_DIR, "/word/*.docx"))
```

--------------------------------

### Determine file encodings from filenames

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Parses filenames to extract encoding information and validates them against the system's supported encodings using iconvlist.

```R
# get encoding from filename
filenames <- list.files(FILEDIR, "^(Indian|UDHR_).*\\.txt$")

head(filenames)
## [1] "IndianTreaty_English_UTF-16LE.txt"  "IndianTreaty_English_UTF-8-BOM.txt"
## [3] "UDHR_Arabic_ISO-8859-6.txt"         "UDHR_Arabic_UTF-8.txt"             
## [5] "UDHR_Arabic_WINDOWS-1256.txt"       "UDHR_Chinese_GB2312.txt"

# Strip the extension
filenames <- gsub(".txt$", "", filenames)
parts <- strsplit(filenames, "_")
fileencodings <- sapply(parts, "[", 3)

head(fileencodings)
## [1] "UTF-16LE"     "UTF-8-BOM"    "ISO-8859-6"   "UTF-8"        "WINDOWS-1256"
## [6] "GB2312"

# Check whether certain file encodings are not supported
notAvailableIndex <- which(!(fileencodings %in% iconvlist()))
fileencodings[notAvailableIndex]
## [1] "UTF-8-BOM"
```

--------------------------------

### Function signature for basename_unique

Source: https://readtext.quanteda.io/reference/basename_unique.html

The function signature for basename_unique, showing the x argument for file paths and the path_only logical flag.

```R
basename_unique(x, path_only = FALSE)
```

--------------------------------

### Check unavailable iconv conversions

Source: https://readtext.quanteda.io/reference/data_files_encodedtexts.html

Identifies which file encodings are not available for conversion on the current platform using iconvlist(). The output shows the message 'Encoding conversions not available for this platform:'.

```r
# find out which conversions are unavailable (through iconv())
cat("Encoding conversions not available for this platform:")
#> Encoding conversions not available for this platform:
notAvailableIndex <- which(!(fileencodings %in% iconvlist()))
#> Error: object 'fileencodings' not found
fileencodings[notAvailableIndex]
#> Error: object 'fileencodings' not found
```

--------------------------------

### Read text files with specified encoding and docvars

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Imports the text files using the previously determined encodings and extracts document variables from the filenames.

```R
txts <- readtext(paste0(DATA_DIR, "/data_files_encodedtexts.zip"), 
                 encoding = fileencodings,
                 docvarsfrom = "filenames", 
                 docvarnames = c("document", "language", "input_encoding"))
print(txts, n = 50)
```

--------------------------------

### readtext()

Source: https://readtext.quanteda.io/reference/index.html

Reads one or more text files into a data.frame with document-level variables. This is the main function for importing text data.

```APIDOC
## readtext()

### Description
Read a text file(s) and return a data.frame with document-level variables.

### Usage
readtext(file, ...)

### Arguments
- **file** (character) - Required - Path to the file(s) to read.
- **...** (optional) - Additional arguments passed to underlying read functions.

### Value
A data.frame with columns: doc_id, text, and any additional document-level variables.

### Examples
```r
# Read a single text file
readtext("path/to/file.txt")

# Read multiple files with a pattern
readtext("path/to/files/*.txt")
```
```

--------------------------------

### docnames(x)

Source: https://readtext.quanteda.io/reference/docnames.html

Returns document names from a readtext object. The function takes a readtext object and returns a character vector of its document names.

```APIDOC
## docnames(x)

### Description
Returns document names from a readtext object.

### Function Signature
```
docnames(x)
```

### Arguments
- **x** - a readtext object whose document names will be extracted

### Value
A character vector of document names.
```

--------------------------------

### Remove page numbers formatted as - X -

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Demonstrates removing page numbers formatted as '- X -' from a text string using regex replacement.

```R
sample_text_b <- "The quick brown fox named Seamus 
- 1 - 
jumps over the lazy dog also named Seamus, with 
- 2 - 
the newspaper from a boy named quick Seamus, in his mouth. 
- 33 - 
The quicker brown fox jumped over 2 lazy dogs."

sample_text_b
## [1] "The quick brown fox named Seamus \n- 1 - \njumps over the lazy dog also named Seamus, with \n- 2 - \nthe newspaper from a boy named quick Seamus, in his mouth. \n- 33 - \nThe quicker brown fox jumped over 2 lazy dogs."

sample_text_b2 <- unlist(stri_split_fixed(sample_text_b, '\n'), use.names = FALSE)
sample_text_b2 <- stri_replace_all_regex(sample_text_b2, "[-] \\d* [-]", "")
sample_text_b2 <- stri_trim_both(sample_text_b2)
sample_text_b2 <- sample_text_b2[sample_text_b2 != '']
stri_paste(sample_text_b2, collapse = '\n')
## [1] "The quick brown fox named Seamus\njumps over the lazy dog also named Seamus, with\nthe newspaper from a boy named quick Seamus, in his mouth.\nThe quicker brown fox jumped over 2 lazy dogs."
```

--------------------------------

### basename_unique(x, path_only = FALSE)

Source: https://readtext.quanteda.io/reference/basename_unique.html

Returns unique basenames from a character vector of file paths. If path_only is TRUE, only the unique part of the path is returned.

```APIDOC
## basename_unique(x, path_only = FALSE)

### Description
Returns unique basenames from a character vector of file paths. If `path_only` is `TRUE`, only the unique part of the path is returned.

### Usage
```r
basename_unique(x, path_only = FALSE)
```

### Arguments
- **x** (character vector) - Required - File paths.
- **path_only** (logical) - Optional - If `TRUE`, only return the unique part of the path.

### Examples
```r
files <- c("../data/glob/subdir1/test.txt", "../data/glob/subdir2/test.txt")
readtext:::basename_unique(files)
#> [1] "subdir1" "subdir2"
# [1] "subdir1/test.txt" "subdir2/test.txt"
readtext:::basename_unique(files, path_only = TRUE)
#> [1] "subdir1" "subdir2"
# [1] "subdir1" "subdir2"
readtext:::basename_unique(c("../data/test1.txt", "../data/test2.txt"))
#> [1] "test1.txt" "test2.txt"
# [1] "test1.txt" "test2.txt"
```
```

--------------------------------

### readtext()

Source: https://readtext.quanteda.io/reference/readtext.html

Reads texts and associated document-level meta-data from one or more source files. Supports various file formats and can extract docvars from filenames or file contents.

```APIDOC
## readtext(file, ignore_missing_files = FALSE, text_field = NULL, docid_field = NULL, docvarsfrom = c("metadata", "filenames", "filepaths"), dvsep = "_", docvarnames = NULL, encoding = NULL, source = NULL, cache = TRUE, verbosity = readtext_options("verbosity"), ...)

### Description
Reads texts and (if any) associated document-level meta-data from one or more source files. The text source files come from the textual component of the files, and the document-level metadata ("docvars") come from either the file contents or filenames.

### Usage
```r
readtext(
  file,
  ignore_missing_files = FALSE,
  text_field = NULL,
  docid_field = NULL,
  docvarsfrom = c("metadata", "filenames", "filepaths"),
  dvsep = "_",
  docvarnames = NULL,
  encoding = NULL,
  source = NULL,
  cache = TRUE,
  verbosity = readtext_options("verbosity"),
  ...
)
```

### Arguments
- **file** (character) - Required - The complete filename(s) to be read. Can be a glob-type wildcard value, a URL, or an archive file. Supported file types include: txt, json, csv, tab, tsv, html, xml, pdf, odt, doc, docx, rtf, zip, tar, tar.gz, tar.bz.
- **ignore_missing_files** (logical) - Optional - If FALSE, an error is thrown when the file argument doesn't resolve to an existing file. Defaults to FALSE.
- **text_field** (character/numeric) - Optional - A variable (column) name or column number indicating where to find the texts. Required for .csv, .json, and .xls/.xlsx files. For XML files, an XPath expression can be specified.
- **docid_field** (character/numeric) - Optional - A variable (column) name or column number indicating where to find document identifiers.
- **docvarsfrom** (character) - Optional - Specifies whether docvars should be taken from filenames, filepaths, or metadata. Defaults to c("metadata", "filenames", "filepaths").
- **dvsep** (character) - Optional - Separator (regular expression) used in filenames to delimit docvar elements when docvarsfrom is "filenames" or "filepaths". Defaults to "_".
- **docvarnames** (character vector) - Optional - Character vector of variable names for docvars. If not used, default names (docvar1, docvar2, ...) are used.
- **encoding** (character vector) - Optional - Either the encoding of all files, or one encoding for each file.
- **source** (character) - Optional - Used to specify specific formats of some input file types, such as "twitter" for JSON and "nexis" for HTML.
- **cache** (logical) - Optional - If TRUE, save remote file to a temporary folder. Only used when file is a URL. Defaults to TRUE.
- **verbosity** (integer) - Optional - Controls the level of output. Defaults to readtext_options("verbosity").
- **...** - Optional - Additional arguments passed to underlying readers, e.g., `collapse` for XML files.

### Details
- **Single file formats:**
  - `txt`: Plain text files.
  - `json`: JSON objects, line-delimited JSON, or Twitter stream JSON. The correct format is automatically detected.
  - `csv`, `tab`, `tsv`: Comma- or tab-separated values.
  - `html`: HTML documents, including Nexis-formatted HTML (use `source = "nexis"`).
  - `xml`: XML documents readable by `xml2::read_xml()`. Additional argument `collapse` can be passed via `...` to specify characters used to append text elements.
  - `pdf`: PDF files converted through pdftools.
  - `odt`: Open Document Text files.
  - `doc`, `docx`: Microsoft Word files.
  - `rtf`: Rich Text Files.
- **Multiple files and file types:**
  - Wildcard values: Any valid pathname with a glob expression.
  - URLs: Remote files are downloaded then loaded.
  - Archives: zip, tar, tar.gz, tar.bz. The contained files must be at the top level or in a single directory.
  - Archives, remote URLs, and glob patterns can resolve to any of the other filetypes.

### Value
A data.frame with the texts and docvars, ready to be used with the quanteda package.

```

--------------------------------

### get_nexis_html function signature

Source: https://readtext.quanteda.io/reference/get_nexis_html.html

The function signature for extracting data from Nexis HTML files.

```R
get_nexis_html(path, paragraph_separator = "\n\n", verbosity, ...)
```

--------------------------------

### Remove page numbers from text

Source: https://readtext.quanteda.io/articles/readtext_vignette.html

Demonstrates removing page numbers formatted as 'page X' from a text string using regex replacement.

```R
sample_text_a <- "The quick brown fox named Seamus jumps over the lazy dog also named Seamus, 
page 1 
with the newspaper from a boy named quick Seamus, in his mouth.
page 2
The quicker brown fox jumped over 2 lazy dogs."

sample_text_a
## [1] "The quick brown fox named Seamus jumps over the lazy dog also named Seamus, \npage 1 \nwith the newspaper from a boy named quick Seamus, in his mouth.\npage 2\nThe quicker brown fox jumped over 2 lazy dogs."

# Remove "page" and respective digit
sample_text_a2 <- unlist(stri_split_fixed(sample_text_a, '\n'), use.names = FALSE)
sample_text_a2 <- stri_replace_all_regex(sample_text_a2, "page \\d*", "")
sample_text_a2 <- stri_trim_both(sample_text_a2)
sample_text_a2 <- sample_text_a2[sample_text_a2 != '']
stri_paste(sample_text_a2, collapse = '\n')
## [1] "The quick brown fox named Seamus jumps over the lazy dog also named Seamus,\nwith the newspaper from a boy named quick Seamus, in his mouth.\nThe quicker brown fox jumped over 2 lazy dogs."
```

--------------------------------

### Read CSV with readtext and text_field

Source: https://readtext.quanteda.io/index.html

Reads a CSV file using readtext(), specifying the text field as 'texts'. The output shows a readtext object with 5 documents and 3 docvars.

```r
readtext(paste0(DATA_DIR, "/csv/inaugCorpus.csv"), text_field = "texts")
```
Tokens:
Raw