### Install dtagger from GitHub Source: https://github.com/timmarchand/dtagger/blob/master/README.md Installs the development version of the dtagger package directly from GitHub using the devtools package. This requires the devtools package to be installed first. ```r # install.packages("devtools") devtools::install_github("timmarchand/dtagger") ``` -------------------------------- ### Get Complete Tag Table with add_tag_tbl() in R Source: https://context7.com/timmarchand/dtagger/llms.txt Generates a comprehensive data frame containing Universal Dependencies annotations, Stanford tags, and MDA tags with full syntactic information. This function is useful for detailed linguistic analysis by consolidating various annotation types. ```r # Process text and get complete annotation table text <- "This is a sample sentence for comprehensive linguistic analysis." result <- add_tag_tbl(text) result # # A tibble: 10 × 19 # doc_id paragraph_id sentence_id token_id token lemma upos xpos feats head_token_id dep_rel # # 1 doc1 1 1 1 This this DET DT Num=S… 6 nsubj ``` -------------------------------- ### Quick Concordancing with quick_conc (R) Source: https://github.com/timmarchand/dtagger/blob/master/README.md The quick_conc function provides a lightweight way to perform concordancing, returning keywords in context (KWIC) in a tidy format. It accepts a character vector of tokenized strings or a single string, along with a character vector of regex patterns or a numeric vector to specify matches. Optional arguments control the number of context tokens and whether to tokenize or separate the text. The output is a tibble with contextualized text around the matched nodes. ```r quick_conc(x, index, n = 5, tokenize = FALSE, separated = FALSE) ``` ```r x <- c("The", "cat", "sat", "on", "the", "mat") index <- c("cat", "sat") quick_conc(x, index, n = 2) ``` ```r x <- "The dog barked loudly, alerting the neighbors of potential danger. A nearby park seemed like the perfect spot for the dog and it quickly made its way there." quick_conc(x, index = "dog", n = 3, tokenize = TRUE, separated = TRUE) quick_conc(x, index = c(4,8,12), tokenize = TRUE) ``` -------------------------------- ### Basic Concordancing with quick_conc Source: https://context7.com/timmarchand/dtagger/llms.txt Generates keyword-in-context (KWIC) concordance lines from tokenized text or raw text. `quick_conc` accepts tokenized input (character vectors) or raw text, allowing specification of search terms by character patterns or numeric indices. It offers customizable context window sizes (`n`) and options for tokenization and separated context columns. ```r # Basic concordancing with tokenized input x <- c("The", "cat", "sat", "on", "the", "mat") index <- c("cat", "sat") quick_conc(x, index, n = 2) # Concordance on raw text with tokenization x <- "The dog barked loudly, alerting the neighbors of potential danger. A nearby park seemed like the perfect spot for the dog and it quickly made its way there." quick_conc(x, index = "dog", n = 3, tokenize = TRUE, separated = FALSE) # Separated context columns for detailed analysis quick_conc(x, index = "dog", n = 3, tokenize = TRUE, separated = TRUE) # Use numeric indices instead of patterns quick_conc(x, index = c(4, 8, 12), tokenize = TRUE, n = 2) # Larger context window words <- c("I", "really", "love", "the", "beautiful", "weather", "today") quick_conc(words, index = "beautiful", n = 4) ``` -------------------------------- ### Initialize UDPipe Model with init_udpipe_model() Source: https://context7.com/timmarchand/dtagger/llms.txt This function downloads and loads a Universal Dependencies language model into the R global environment. It enables text annotation tasks such as Part-of-Speech tagging and dependency parsing using functions like add_st_tags. The model persists for the R session, and different language models can be initialized. ```r # Initialize default English model (english-ewt) init_udpipe_model() # Now you can use add_st_tags and related functions text <- "The model is ready to use." tagged <- add_st_tags(text) # Initialize different language model init_udpipe_model(lang = "french") french_text <- "Le chat dort sur le tapis." french_tagged <- add_st_tags(french_text) # Initialize specific English variant init_udpipe_model(lang = "english-gum") # GUM corpus model # Alternative: english-lines, english-partut # Spanish model init_udpipe_model(lang = "spanish") spanish_tagged <- add_st_tags("El gato duerme en la alfombra.") # German model init_udpipe_model(lang = "german") german_tagged <- add_st_tags("Die Katze schläft auf dem Teppich.") # The model persists for the entire R session # No need to reinitialize unless switching languages text1 <- "First sentence." text2 <- "Second sentence." tagged1 <- add_st_tags(text1) # uses existing model tagged2 <- add_st_tags(text2) # uses existing model # Check if model exists before using if (exists("udmodel")) { tagged <- add_st_tags("Text to process") } else { init_udpipe_model() tagged <- add_st_tags("Text to process") } ``` -------------------------------- ### Analyze Directory and Calculate MDA Scores with dtagger Source: https://context7.com/timmarchand/dtagger/llms.txt Processes a directory of text files, applies linguistic tagging, and calculates multi-dimensional analysis (MDA) scores. It requires initialization of the Universal Dependencies model using `init_udpipe_model()` and can optionally exclude specific tags or analyze a sample of files. ```r library(dtagger) # Initialize the Universal Dependencies model (required for tagging) init_udpipe_model() # Analyze a directory structure like: # corpus_folder/ # $$fiction/ # text1.txt # text2.txt # $$academic/ # text3.txt # text4.txt results <- dtag_directory( path = "corpus_folder", n = NULL, # analyze all files (or set maximum number) ST = FALSE, # texts don't have Stanford tags yet deflated = TRUE # include deflated dimension scores (removes rare features) ) # Access different result components print(results$corpus_dimension_scores) # # A tibble: 2 × 9 # corpus corpus_text_type most_common_text_type Dimension1 Dimension2 Dimension3 Dimension4 Dimension5 Dimension6 # # 1 fiction face-to-face face-to-face 15.2 -2.1 1.4 -0.8 2.3 -1.1 # 2 academic academic_prose academic_prose -12.5 3.8 -2.1 0.9 -1.5 0.7 print(results$document_dimension_scores) # # A tibble: 4 × 10 # corpus doc_id Dimension1 Dimension2 Dimension3 Dimension4 Dimension5 Dimension6 closest_text_type dimension_tags # # 1 fiction text1 14.8 -2.3 1.5 -0.7 2.1 -1.0 telephone # 2 fiction text2 15.6 -1.9 1.3 -0.9 2.5 -1.2 face-to-face # View linguistic feature counts for a specific document print(head(results$dimension_tags %>% filter(doc_id == "text1") %>% select(feature, detail, count, value, dscore))) # # A tibble: 6 × 5 # feature detail count value dscore # # 1 first_person_pronouns 45 4.12 1.85 # 2 contractions 32 2.93 1.42 # 3 private_verbs 28 2.56 1.21 # View tagged text with both ST and MDA tags print(head(results$tokenized_tags)) # # A tibble: 3 × 4 # corpus doc_id st mda # # 1 fiction text1 "I_PPSS" "I_PPSS" # 2 fiction text1 "don't_XX" "don't_XX" # 3 fiction text1 "know_VB" "know_VB" # Statistical comparison between corpora (Tukey HSD test) print(results$Tukey_hsd) # # A tibble: 6 × 7 # dimension contrast null.value estimate conf.low conf.high p.value # # 1 Dimension1 academic-fiction 0 -27.7 -31.2 -24.2 <0.001 # 2 Dimension2 academic-fiction 0 5.9 3.8 8.0 <0.001 # Exclude manually-checked tags or TTR from analysis results_filtered <- dtag_directory( path = "corpus_folder", exclude = c("", "") # removes 11 tags requiring manual verification ) # Analyze only first 10 texts with deflated scores only results_sample <- dtag_directory( path = "corpus_folder", n = 10, deflated = FALSE # skip deflated dimension calculations ) ``` -------------------------------- ### Custom Analysis Workflows with dtagger functions Source: https://context7.com/timmarchand/dtagger/llms.txt Allows advanced users to access individual tagging functions for building custom analysis workflows. This enables integration with concordancing for qualitative validation or feature extraction for machine learning. ```python from dtagger.tags import tag_document, tag_sentence # Example of tagging a single document: document_text = "This is a sample document." results = tag_document(document_text) # Example of tagging a sentence: sentence_text = "This is a sentence." sentence_results = tag_sentence(sentence_text) print(results) print(sentence_results) ``` -------------------------------- ### Add Stanford POS Tags with add_st_tags() in R Source: https://context7.com/timmarchand/dtagger/llms.txt Tokenizes and annotates text with Stanford part-of-speech tags using Universal Dependencies models. Supports handling hesitation markers, custom tokenization (horizontal, vertical), and full syntactic parsing. Requires loading a UDpipe model first. ```r # Basic text tagging text <- "This is an example sentence to be tagged for linguistic analysis." init_udpipe_model() # load model first tagged <- add_st_tags(text) tagged # [1] "This_DT" "is_VBZ" "an_DT" "example_NN" "sentence_NN" "to_TO" "be_VB" "tagged_VBN" # [9] "for_IN" "linguistic_JJ" "analysis_NN" "._." # Tag tokenized speech with hesitation markers speech <- c("I", "don't", "know", "erm", ",", "whether", "to", "include", "hesitation", "markers", ".") tagged_speech <- add_st_tags( speech, st_hesitation = TRUE, # extract and tag hesitation markers flattened = FALSE # input is already tokenized ) tagged_speech # [1] "I_PPSS" "don't_XX" "know_VB" "erm_UH" "," "whether_CS" "to_TO" "include_VB" # [9] "hesitation_NN" "markers_NNS" "._." # Control tokenization behavior text_hyphenated <- "I'm in a part-time job, at the moment." # Default tokenization (may split hyphenated words) add_st_tags(text_hyphenated) # [1] "I_PRP" "'m_VBP" "in_IN" "a_DT" "part_NN" "-_HYPH" "time_NN" "job_NN" ... # Horizontal tokenization (whitespace-based, preserves user tokenization) add_st_tags(text_hyphenated, tokenizer = "horizontal") # [1] "I'm_PRP" "in_IN" "a_DT" "part-time_JJ" "job_NN" ",_," "at_IN" "the_DT" "moment_NN" "._." # Vertical tokenization for multi-word expressions (newline-separated input) text_vertical <- "I'm\nin\na\npart-time\njob\n,\n\nat the moment\n.\n" add_st_tags(text_vertical, tokenizer = "vertical") # [1] "I'm_PRP" "in_IN" "a_DT" "part-time_JJ" "job_NN" ",_," "at the moment_NP" "._." # Get full Universal Dependencies parse tree (not just POS tags) full_parse <- add_st_tags( "The cat sat on the mat.", skip_parse = FALSE # return complete UD annotation ) full_parse # # A tibble: 7 × 17 # doc_id paragraph_id sentence_id token_id token lemma upos xpos st # # 1 doc1 1 1 1 The the DET DT The_DT # 2 doc1 1 1 2 cat cat NOUN NN cat_NN # 3 doc1 1 1 3 sat sit VERB VBD sat_VBD # ... with 8 more variables: feats, head_token_id, dep_rel, deps, misc # Use different language model init_udpipe_model(lang = "french") # download and load French model french_tagged <- add_st_tags("Le chat dort sur le tapis.") ``` -------------------------------- ### Add Multi-Dimensional Analysis Tags with add_mda_tags() in R Source: https://context7.com/timmarchand/dtagger/llms.txt Applies 67 linguistic feature tags to Stanford-tagged text based on Biber's MDA framework. Identifies grammatical and lexical features across six dimensions of linguistic variation. Can optionally handle hesitation markers and display progress for long texts. ```r # Simple example with pre-tagged tokens text <- c("I_PPSS", "have_VB", "a_DET", "dog_NN") mda_tagged <- add_mda_tags(text) mda_tagged # [1] "I_PPSS" "have_VB" "a_DET" "dog_NN" # Complete workflow: raw text to full MDA tagging text <- "This example is short and sweet. This means that not all the tags will have been included, which is why this is really only a guide and it should be used with that in mind. Otherwise, I think it may lead to disappointment." init_udpipe_model() st_tagged <- add_st_tags(text) mda_tagged <- add_mda_tags(st_tagged) # View results (showing sample of tagged output) head(mda_tagged, 15) # [1] "This_DT" "example_NN" "is_VBZ" "short_JJ" "and_CC" # [6] "sweet_JJ" "._." "This_DT" "means_VBZ" "that_IN" # [11] "not_RB" "all_DT" "the_DT" "tags_NNS" "will_MD" # Control hesitation marker handling speech <- c("Well_UH", "I_PPSS", "think_VB", "uh_UH", "maybe_RB") # With hesitation extraction (experimental) mda_with_hesitation <- add_mda_tags(speech, mda_hesitation = TRUE) # Without hesitation extraction mda_no_hesitation <- add_mda_tags(speech, mda_hesitation = FALSE) # Enable progress messages for long texts large_text <- add_st_tags(paste(rep(text, 100), collapse = " ")) mda_tracked <- add_mda_tags(large_text, progress = TRUE) # Adding MDA tags to [large_text content preview]... ``` -------------------------------- ### Add Stanford _ST Tags with `add_st_tags` (R) Source: https://github.com/timmarchand/dtagger/blob/master/README.md The `add_st_tags` function annotates text using the Universal Dependencies (UD) model via the udpipe package. It can tokenize and tag text with Stanford POS tags and extract hesitation markers. It requires the 'udpipe' package and the 'udmodel' object in the global environment. ```r # Example text: text <- "This is an example sentence to be tagged" # Example speech, tokenized: speech <- c("I","don't", "know" , "erm" ,",", "whether" , "to" , "include" ,"hesitation" , "markers", ".") # Initiate udpipe model init_udpipe_model() # Tag text add_st_tags(text) # Tag speech add_st_tags(speech, st_hesitation = TRUE, tokenized = TRUE) ``` -------------------------------- ### Concordancing by Tags with conc_by_tag (R) Source: https://github.com/timmarchand/dtagger/blob/master/README.md The conc_by_tag function enables fine-grained concordance searches on tagged text, commonly used with output from udpipe::udpipe_annotate and dtagger functions. It takes a relational data frame as input and allows matching based on up to two tag columns (e.g., 'xpos' and 'dep_rel'). The function returns a data frame in KWIC format, showing the matched keywords in context. It can be used with specified columns for text, tags, and identifying details, and accepts additional arguments passed to dtagger::quick_conc. ```r conc_by_tag( data, what = "token", tag = "xpos", match = "^JJ$", cols = c("doc_id", "lemma"), tag2 = "dep_rel", match2 = "^amod$" ) ``` ```r conc_by_tag( data, what = "token", tag = "xpos", match = "JJ", cols = c("doc_id", "dep_rel"), separated = TRUE, n = 3 ) ``` ```r # Process text with the add_tag_tbl function (assuming add_mda_tags function is defined) text <- c(doc1 = "This is a simple sentence with a specific keyword.", doc2 = "Is this one more complex or simpler?") data <- add_tag_tbl(text) ``` -------------------------------- ### Batch Processing Directory with dtagger Source: https://context7.com/timmarchand/dtagger/llms.txt Automates the processing of text corpora organized in a directory structure. Assumes documents are organized with prefixes for corpus identification. Outputs tidy data frames compatible with tidyverse workflows. ```python from dtagger import dtag_directory # Example usage: dtag_directory( input_path="path/to/your/corpus/directory", output_path="path/to/output/results.csv" ) ``` -------------------------------- ### Add MDA Tags to Text Files Source: https://github.com/timmarchand/dtagger/blob/master/README.md A core function within the dtagger package used to add MDA tags to text files. This function is modular, allowing for inspection and modification of the underlying algorithms for tag generation. ```r add_mda_tags() ``` -------------------------------- ### Perform Key Word in Context (KWIC) Concordancing Source: https://github.com/timmarchand/dtagger/blob/master/README.md Provides a utility function for performing fast Key Word in Context (KWIC) concordancing on text data. This is useful for analyzing the immediate context surrounding specific words or phrases. ```r quick_conc() ``` -------------------------------- ### Concordancing by Tag Matches Source: https://github.com/timmarchand/dtagger/blob/master/README.md Displays Key Word in Context (KWIC) results specifically for search results that match given tags. This function helps in analyzing concordances based on the presence of specific MDA tags within the text. ```r conc_by_tag() ``` -------------------------------- ### Tag Directory of Text Files with dtag_directory Source: https://github.com/timmarchand/dtagger/blob/master/README.md The main function to tag a directory of plain text files using MDA tags. It analyzes text files, calculates dimension scores based on Biber’s (1988) standards, and returns a list of tibbles containing tagged texts, scores, and word counts. Optional arguments control the number of files analyzed, presence of _ST tags, and deflation of rare features. ```r library(dtagger) ## basic example code dtag_directory(path = "path_to_folder", n = NULL, ST = FALSE, deflated = TRUE) ``` -------------------------------- ### Add Tag Table with `add_tag_tbl` (R) Source: https://github.com/timmarchand/dtagger/blob/master/README.md The `add_tag_tbl` function acts as a wrapper for `add_st_tags` and `add_mda_tags`, while also preserving all Universal Dependency tags generated by the udpipe model. It processes input text and returns a comprehensive table of tags. ```r # Process text with the add_tag_tbl function text <- "This is a sample sentence." result <- add_tag_tbl(text) ``` -------------------------------- ### Compare Tagging Output with missing_tags Function in R Source: https://github.com/timmarchand/dtagger/blob/master/README.md Demonstrates the usage of the `missing_tags` function to compare tagging results between two R character vectors. The function helps identify missing tags at specific token positions based on provided regular expressions. Input vectors must be of equal length and token-aligned. ```r # create two example vectors # vec1 tagged with ewt udpipe English model vec1 <- c("This_DT", "is_VBZ", "a_DT", "test_NN", "sentence_NN", "with_IN", "tags_NNS", "._.") # vec2 tagged with line udpipe English model vec2 <- c("This_DEM-SG", "is_PRES", "a_IND-SG", "test_SG-NOM", "sentence_SG-NOM", NA, "tags_PL-NOM", "._Period") # find tags in vec1 that are missing in vec2 missing_tags(vec1, vec2, regex = "DT") # find tags in vec2 that are missing in vec1 missing_tags(vec2, vec1, regex = "Period") # find tags in vec1 that are missing in vec2 but with different tag searches missing_tags(vec1, vec2, regex1 = "DT", regex2 = "IND-SG") ``` -------------------------------- ### Add MDA Tags with `add_mda_tags` (R) Source: https://github.com/timmarchand/dtagger/blob/master/README.md The `add_mda_tags` function adds tags to a character vector of text, assuming the input is already tokenized and tagged with Stanford POS tags. It applies tagging rules and can optionally extract hesitation markers. Basic usage involves passing the tagged text to the function. ```r # Basic example: text <- c("I_PPSS", "have_VB", "a_DET", "dog_NN") add_mda_tags(text) # Detailed example: # Generate some text text <- "This example is short and sweet. This means that not all the tags will have been included, which is why this is really only a guide and it should be used with that in mind. Otherwise, I think it may lead to disappointment." # Load udpipe model into the global environment for _ST tagging init_udpipe_model() # Add Stanford tags to text text <- add_st_tags(text) # Add tags add_mda_tags(text) ``` -------------------------------- ### Concordance by Tag with conc_by_tag Source: https://context7.com/timmarchand/dtagger/llms.txt Searches annotated text data (typically from `add_tag_tbl`) for specific tag patterns and displays matches in KWIC format. `conc_by_tag` supports filtering based on one or two tag types (`tag`, `tag2`) with corresponding match patterns (`match`, `match2`), enabling precise linguistic feature identification. Additional columns can be included in the output. ```r # Prepare tagged data text <- c( doc1 = "This is a simple sentence with a specific keyword.", doc2 = "Is this one more complex or simpler?" ) init_udpipe_model() data <- add_tag_tbl(text) # Find all adjectives in attributive modifier position conc_by_tag( data, what = "token", # show tokens in concordance tag = "xpos", # first filter: Stanford POS tag match = "^JJ$", # match adjectives cols = c("doc_id", "lemma"), # additional columns to include tag2 = "dep_rel", # second filter: dependency relation match2 = "^amod$" # match attributive modifiers only ) # Find all adjectives with separated context and dependency info conc_by_tag( data, what = "token", tag = "xpos", match = "JJ", # partial match (includes JJR, JJS) cols = c("doc_id", "dep_rel"), separated = TRUE, n = 3 ) ``` -------------------------------- ### Search MDA Tags Directly Source: https://context7.com/timmarchand/dtagger/llms.txt This function searches for specific MDA tags within a dataset. It allows filtering by tag type, match patterns, and specifying which columns to return. Useful for extracting specific linguistic information like adjectives or pronouns in subject positions. ```r data %> conc_by_tag( what = "token", tag = "mda", match = "", # find MDA adjective tags cols = c("doc_id", "st", "mda") ) data %> conc_by_tag( what = "lemma", # show lemmas instead of surface forms tag = "mda", match = "", # first person pronouns tag2 = "dep_rel", match2 = "nsubj", # nominal subject cols = c("doc_id", "token") ) data %> conc_by_tag( what = "token", tag = "upos", # Universal POS match = "VERB", tag2 = "mda", match2 = "PRIV", # private verbs (think, believe, feel) cols = c("doc_id", "lemma", "dep_rel"), n = 4 ) ``` -------------------------------- ### Tagging Text with add_tag_tbl Source: https://context7.com/timmarchand/dtagger/llms.txt Tags text data using `add_tag_tbl`, which can process single strings or vectors of strings. It supports different tokenization methods, such as 'horizontal'. The function returns a tibble with detailed annotations for each token, including POS tags, dependency relations, and other linguistic features. ```r texts <- c( doc1 = "First document with simple text.", doc2 = "Second document is slightly longer and more complex." ) multi_result <- add_tag_tbl(texts) # Filter to see only MDA-tagged features multi_result %>% filter(str_detect(mda, "<")) %>% select(doc_id, token, st, mda, upos, dep_rel) # Pass additional arguments to add_st_tags result_horiz <- add_tag_tbl( "Multi-word test", tokenizer = "horizontal" ) ``` -------------------------------- ### Compare Tag Outputs with missing_tags() Source: https://context7.com/timmarchand/dtagger/llms.txt The missing_tags function identifies discrepancies between two sets of tagged text. It can compare tags based on regular expressions, highlighting patterns present in one vector but absent in the other at corresponding positions. This is useful for evaluating the consistency of different annotation models or manual vs. automatic tagging. ```r # Compare tags from different udpipe models # vec1 tagged with ewt udpipe English model vec1 <- c("This_DT", "is_VBZ", "a_DT", "test_NN", "sentence_NN", "with_IN", "tags_NNS", "._.") # vec2 tagged with lines udpipe English model vec2 <- c("This_DEM-SG", "is_PRES", "a_IND-SG", "test_SG-NOM", "sentence_SG-NOM", NA, "tags_PL-NOM", "._Period") # Find determiners (DT) in vec1 that are missing in vec2 missing_tags(vec1, vec2, regex = "DT") # Find Period tags in vec2 missing from vec1 missing_tags(vec2, vec1, regex = "Period") # Different regex patterns for each vector missing_tags( vec1, vec2, regex1 = "DT", # search for DT in vec1 regex2 = "IND-SG" # but require IND-SG in vec2 ) # Compare manual vs automatic MDA tagging auto_tagged <- add_st_tags("I think that dogs are wonderful pets.") %>% add_mda_tags() manual_tagged <- c("I_PPSS", "think_VB", "that_IN", "dogs_NNS", "are_VBR", "wonderful_JJ", "pets_NNS", "._.") # Find private verb tags in manual that auto-tagger missed missing_tags(manual_tagged, auto_tagged, regex = "") # Find any MDA tags in auto output not in manual missing_tags(auto_tagged, manual_tagged, regex = "<[A-Z]+>") ``` -------------------------------- ### Identify Missing Tags Between Taggers Source: https://github.com/timmarchand/dtagger/blob/master/README.md A utility function designed to identify tags that are present in the output of one tagger but are missing in the output of another. This is useful for comparing and validating different tagging methodologies. ```r missing_tags() ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.