### Tagging sentences with Hanover Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Demonstrates how to tokenize a sentence and use the tagger to retrieve POS tags and lemmata. Includes examples of varying the taglevel parameter to control the granularity of the output. ```python import nltk from pprint import pprint sent = "Die Europawahl in den Niederlanden findet immer donnerstags statt." words = nltk.word_tokenize(sent) lemmata = tagger.tag_sent(words) pprint(lemmata) ``` ```python sent = "Die Sozialdemokraten haben ersten Prognosen zufolge die Europawahl in den Niederlanden gewonnen." words = nltk.word_tokenize(sent) lemmata = tagger.tag_sent(words, taglevel=3) pprint(lemmata) ``` ```python sent = "Der palästinensische Schriftsteller Emil Habibi ist der einzige Autor im Nahen Osten, dessen Werk von allen Seiten größte und offizielle Anerkennung zuteil geworden ist." words = nltk.word_tokenize(sent) tags = tagger.tag_sent(words, taglevel=0) print(tags) ``` -------------------------------- ### Initialize HanTa German Tagger Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb Initializes the HanTa German tagger by importing the necessary library and loading the German morphological model. This setup is crucial for all subsequent tagging and analysis operations. ```python import sys sys.path.insert(1, '..') import HanoverTagger as ht tagger = ht.HanoverTagger(r'../morphmodel_ger.pgz') ``` -------------------------------- ### Install and Import HanTa Source: https://context7.com/wartaal/hanta/llms.txt Instructions for installing the HanTa package via pip and importing the core HanoverTagger class into your Python environment. ```python !pip install HanTa from HanTa import HanoverTagger as ht ``` -------------------------------- ### List German POS Tags with Examples using HanTa Source: https://context7.com/wartaal/hanta/llms.txt Displays all available Part-of-Speech (POS) tags within the German language model of HanTa, along with example words for each tag. This function helps in understanding the tagset used by the library. ```python from HanTa import HanoverTagger as ht tagge r = ht.HanoverTagger('morphmodel_ger.pgz') # Print all POS tags with random examples from the training data tagge r.list_postags() ``` -------------------------------- ### GET /list_postags Source: https://context7.com/wartaal/hanta/llms.txt Retrieves a list of all Part-of-Speech (POS) tags available in the currently loaded model, along with example words for each tag. ```APIDOC ## GET /list_postags ### Description Displays all POS tags defined in the model with example words extracted from the training data. ### Method GET ### Endpoint /list_postags ### Response #### Success Response (200) - **tags** (dict) - A dictionary where keys are POS tags and values are lists of example words. #### Response Example { "NN": ["regierung", "koalition", "kosten"], "VV(FIN)": ["verspricht", "fand", "stehe"] } ``` -------------------------------- ### List German Morpheme Tags with Examples using HanTa Source: https://context7.com/wartaal/hanta/llms.txt Lists all morpheme tags utilized by the German language model in HanTa, providing example words for each. This is useful for detailed morphological analysis and understanding word structure. ```python from HanTa import HanoverTagger as ht tagge r = ht.HanoverTagger('morphmodel_ger.pgz') # Print all morpheme tags with examples tagge r.list_mtags() ``` -------------------------------- ### Analyze Words and Sentences Source: https://github.com/wartaal/hanta/blob/master/README.md Shows how to retrieve POS tags and lemmas for individual words and full sentences. Includes examples of basic analysis and advanced morphological breakdown using taglevel parameters. ```python tagger_en.tag_word('eating') tagger_en.analyze('unhappiest') tagger_nl.analyze('huishoudhulpje', taglevel=3) ``` -------------------------------- ### Analyze Dutch Words and Sentences with HanTa Source: https://context7.com/wartaal/hanta/llms.txt Provides examples for analyzing Dutch words and sentences using the HanTa library. Demonstrates detailed morphology analysis for compound words, disambiguation of homographs using POS tags, and sentence tagging. ```python from HanTa import HanoverTagger as ht import nltk tagge r_nl = ht.HanoverTagger('morphmodel_dutch.pgz') # Analyze compound words with detailed morphology print(tagger_nl.analyze('huishoudhulpje', taglevel=1)) print(tagger_nl.analyze('huishoudhulpje', taglevel=3)) # Disambiguate homographs using pos parameter print(tagger_nl.analyze('staat', taglevel=2, pos='WW(pv,tgw,met-t)')) print(tagger_nl.analyze('staat', taglevel=2, pos='N(soort,ev,basis,zijd,stan)')) # Tag Dutch sentences sent = "Elk jaar wisselen ruim 1 miljoen Nederlanders van zorgverzekeraar." words = nltk.word_tokenize(sent) lemmata = tagger_nl.tag_sent(words, taglevel=1) print(lemmata) ``` -------------------------------- ### Perform Probabilistic POS Tagging Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Uses the tag_word method to retrieve a list of probable POS tags with their associated log-probabilities. Includes examples of using the cutoff parameter to filter results and the casesensitive parameter to influence tag probability estimation. ```python tagger.tag_word('Angeln') print(tagger.tag_word('verdachte', cutoff=10)) tagger.tag_word('angeln', casesensitive=False) tagger.tag_word('angeln', casesensitive=True) ``` -------------------------------- ### List POS-tags and Morpheme tags Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb These methods allow the user to retrieve and inspect the available Part-of-Speech (POS) tags and morpheme tags used by the HanTa tagger, including random examples for each category. ```python tagger.list_postags() tagger.list_mtags() ``` -------------------------------- ### Load Morphological Data from CSV (Python) Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb This script reads morphological data from a CSV file named 'labeledmorph_ger.csv' using UTF-8 encoding. It filters out lines that start with '-1' and stores the valid lines in a list. The primary input is the file path, and the output is a list of strings representing the filtered data. ```python import codecs datafile = codecs.open(r"labeledmorph_ger.csv", "r","utf-8") morphdata = [] for line in datafile: if not line.startswith('-1'): morphdata.append(line) ``` -------------------------------- ### Load and Filter Morphological Data Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb This Python script reads a CSV file containing morphological data, specifically 'labeledmorph_en.csv'. It uses the 'codecs' module for UTF-8 encoding and filters out lines that start with '-1', presumably indicating invalid or header entries. The filtered data is stored in the 'morphdata' list. ```python import codecs datafile = codecs.open(r"labeledmorph_en.csv", "r","utf-8") morphdata = [] for line in datafile: if not line.startswith('-1'): morphdata.append(line) ``` -------------------------------- ### Initialize HanTa Language Models Source: https://github.com/wartaal/hanta/blob/master/README.md Demonstrates how to load specific language models for German, Dutch, and English using the HanoverTagger class. ```python from HanTa import HanoverTagger as ht tagger_de = ht.HanoverTagger('morphmodel_ger.pgz') tagger_nl = ht.HanoverTagger('morphmodel_dutch.pgz') tagger_en = ht.HanoverTagger('morphmodel_en.pgz') ``` -------------------------------- ### Train and Load Custom HanoverTagger Model Source: https://context7.com/wartaal/hanta/llms.txt This snippet demonstrates how to initialize the TrainHanoverTagger, load training data from a tab-separated file, train the model, and save it to a .pgz file. It also shows how to instantiate the tagger with the custom model to perform word analysis. ```python from HanTa.HanoverTagger import TrainHanoverTagger trainer = TrainHanoverTagger() with open('training_data.txt', 'r', encoding='utf-8') as fin: trainer.load(fin) trainer.train_model(observed_values=True) trainer.write_model('custom_model.pgz') from HanTa import HanoverTagger as ht custom_tagger = ht.HanoverTagger('custom_model.pgz') print(custom_tagger.analyze('example_word')) ``` -------------------------------- ### Load Dutch Morphology Model with Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Initializes the Hanta Tagger with a pre-trained Dutch morphology model. This model is loaded from a specified file path. ```python tagger_nl = ht.HanoverTagger('morphmodel_dutch.pgz') ``` -------------------------------- ### Load English Morphology Model with Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Initializes the Hanta Tagger with a pre-trained English morphology model. This model is loaded from a specified file path. ```python tagger_en = ht.HanoverTagger('morphmodel_en.pgz') ``` -------------------------------- ### Load Model and Analyze Words Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Demonstrates loading a pre-trained morphological model and using the analyze method to retrieve POS tags, lemmas, and morphological structures. The taglevel parameter controls the granularity of the output. ```python tagger = ht.HanoverTagger('morphmodel_ger.pgz') print(tagger.analyze('Fachmärkte')) print(tagger.analyze('Fachmärkte', taglevel=3)) print(tagger.analyze('wirft', taglevel=1)) print(tagger.analyze('vertraute', taglevel=3, pos='VV(FIN)')) ``` -------------------------------- ### Initialize HanoverTagger in Python Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb This Python snippet demonstrates how to initialize the HanoverTagger for English. It includes setting up the system path to import the necessary module and loading the English morphological model. ```python import sys sys.path.insert(1, '..') import HanoverTagger as ht tagger = ht.HanoverTagger(r'../morphmodel_en.pgz') ``` -------------------------------- ### Analyze word morphology with Hanta tagger Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Demonstrates how to use the tagger.analyze method to perform morphological decomposition on Dutch words. It returns the lemma and a list of morphological components. ```python tagger.analyze("beïnvloedt", taglevel = 3) tagger.analyze("zandwinningsschepen", taglevel = 3) ``` -------------------------------- ### Analyze word morphology and POS tags Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Demonstrates how to use the tagger.analyze method with different tag levels to retrieve morphological structures and part-of-speech tags for various Dutch words. ```python tagger.analyze('waren', taglevel=1) tagger.analyze('ondergelopene', taglevel=2) tagger.analyze('voorspellingsmogelijkheden', taglevel=3) tagger.analyze('ijsbaantje', taglevel=3, pos='N(soort,ev,dim,onz,stan)') ``` -------------------------------- ### Initialize HanoverTagger for Dutch Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb This Python snippet demonstrates how to initialize the HanoverTagger for Dutch. It imports the necessary module and loads the Dutch language model from a specified path. Ensure the 'morphmodel_dutch.pgz' file is accessible. ```python import sys sys.path.insert(1, '..') import HanoverTagger as ht #Do not import form the package but from the parent folder where the latest source file is found tagger = ht.HanoverTagger(r'../morphmodel_dutch.pgz') ``` -------------------------------- ### Analyze sentences with NLTK Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Shows how to integrate NLTK's word_tokenize with the Hanta tagger to perform part-of-speech tagging on full sentences. ```python import nltk zin = 'De kinderen waren met een voetbal aan het voetballen.' woorden = nltk.word_tokenize(zin) tagger.tag_sent(woorden, taglevel=1) ``` -------------------------------- ### Analyze words with varying tag levels Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Demonstrates how to use the analyze method with different taglevel parameters to retrieve morphological breakdowns and part-of-speech tags for various English words. ```python print(tagger.analyze('walking',taglevel=0)) print(tagger.analyze('walking',taglevel=1)) print(tagger.analyze('walking',taglevel=2)) print(tagger.analyze('walking',taglevel=3)) print(tagger.analyze('walks',taglevel=3)) print(tagger.analyze('wishes',taglevel=3)) print(tagger.analyze('loving',taglevel=3)) print(tagger.analyze('unexpectedly',taglevel=3)) print(tagger.analyze('developed',taglevel=3)) print(tagger.analyze('honestly',taglevel=3)) print(tagger.analyze('men',taglevel=3)) print(tagger.analyze('women',taglevel=2)) print(tagger.analyze('actualities',taglevel=3)) print(tagger.analyze('actualities',taglevel=1)) print(tagger.analyze('simplest',taglevel=3)) print(tagger.analyze('simplest',taglevel=1)) print(tagger.analyze('reliable',taglevel=3)) print(tagger.analyze('heavily',taglevel=3)) print(tagger.analyze('unhappiest',taglevel=3)) print(tagger.analyze('intimidating',taglevel=3)) ``` -------------------------------- ### Process and tag full sentences Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Uses NLTK to tokenize a sentence and then applies the Hanta tagger at various levels to process the entire sequence of words. ```python import nltk sent = "Tackling the entire kitchen can be an intimidating task, so here's a manageable list of things to clean, ingredients to check, equipment to organize and more." words = nltk.word_tokenize(sent) print(tagger.tag_sent(words,taglevel = 0)) print('----') print(tagger.tag_sent(words,taglevel = 1)) print('----') print(tagger.tag_sent(words,taglevel = 2)) print('----') print(tagger.tag_sent(words,taglevel = 3)) ``` -------------------------------- ### Analyze Dutch Sentence Morphology with NLTK and Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Tokenizes a Dutch sentence using NLTK and then analyzes each word using the Hanta Tagger with taglevel=3. This provides detailed lemma and morphological information for the entire sentence. ```python sent = "Elk jaar wisselen ruim 1 miljoen Nederlanders van zorgverzekeraar. " words = nltk.word_tokenize(sent) lemmata = tagger_nl.tag_sent(words,taglevel= 3) pprint(lemmata) ``` -------------------------------- ### Analyze Dutch Word Morphology with Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Analyzes a Dutch word ('staat') to determine its morphological tags and lemmas. It demonstrates using different tag levels and specifying part-of-speech tags for more precise analysis. ```python print(tagger_nl.analyze('staat',taglevel=2,pos='WW(pv,tgw,met-t)')) print(tagger_nl.analyze('staat',taglevel=2,pos='N(soort,ev,basis,zijd,stan)')) ``` -------------------------------- ### Analyze English Sentence Morphology with NLTK and Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Tokenizes an English sentence using NLTK and then analyzes it with the Hanta Tagger at different tag levels (0, 1, 2, 3). This demonstrates how the tagger processes sentences and extracts morphological information. ```python sent = "Tackling the entire kitchen can be an intimidating task, so here's a manageable list of things to clean, ingredients to check, equipment to organize and more." words = nltk.word_tokenize(sent) print(tagger_en.tag_sent(words,taglevel = 0)) print('----') print(tagger_en.tag_sent(words,taglevel = 1)) print('----') print(tagger_en.tag_sent(words,taglevel = 2)) print('----') print(tagger_en.tag_sent(words,taglevel = 3)) ``` -------------------------------- ### Analyze English Words and Sentences with HanTa Source: https://context7.com/wartaal/hanta/llms.txt Demonstrates the use of HanTa for English language processing, including analyzing verb forms at different detail levels, obtaining POS probabilities for ambiguous words, and tagging English sentences. ```python from HanTa import HanoverTagger as ht import nltk tagge r_en = ht.HanoverTagger('morphmodel_en.pgz') # Analyze verb forms print(tagger_en.analyze('walking', taglevel=0)) print(tagger_en.analyze('walking', taglevel=1)) print(tagger_en.analyze('walking', taglevel=3)) # Get POS probabilities for ambiguous words print(tagger_en.tag_word('walks')) # Tag English sentences sent = "Tackling the entire kitchen can be an intimidating task." words = nltk.word_tokenize(sent) tags = tagger_en.tag_sent(words, taglevel=0) print(tags) lemmata = tagger_en.tag_sent(words, taglevel=1) print(lemmata) ``` -------------------------------- ### Analyze English Word Morphology with Varying Tag Levels Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Analyzes the English word 'walking' with different tag levels (0 to 3) to show the extracted morphological information, including base form and suffixes. ```python print(tagger_en.analyze('walking',taglevel=0)) print(tagger_en.analyze('walking',taglevel=1)) print(tagger_en.analyze('walking',taglevel=2)) print(tagger_en.analyze('walking',taglevel=3)) ``` -------------------------------- ### Load Tab-Separated Data for NLP Tasks (Python) Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb This function loads data from lines where each line is tab-separated, expecting fields for sentence number, word, lemma, stem, and tag. It groups words into sentences based on sentence numbers. The primary input is a list of strings, and the output is a list of tuples, where each tuple contains a sentence number and a list of word-lemma-tag tuples. ```python def load(lines): data = [] sent = [] lastsentnr = 1 for line in lines: (sentnr,word,lemma,stem,tag) = line.split('\t')[:5] if sentnr != lastsentnr: if len(sent) > 0: data.append((sentnr,sent)) sent = [] lastsentnr = sentnr sent.append((word,lemma,tag)) return data ``` -------------------------------- ### Compare tag levels for word analysis Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Iterates through different tag levels (0-3) to observe how the granularity of morphological analysis changes for specific words. ```python for level in range(4): print(tagger.analyze('mannen', taglevel=level)) for level in range(4): print(tagger.analyze('verliepen', taglevel=level)) ``` -------------------------------- ### Load and parse morphological training data Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Functions to load tab-separated morphological data from a file and parse it into a structured list of sentences containing words, lemmas, and tags. ```python def load(lines): data = [] sent = [] lastsentnr = 1 for line in lines: (sentnr,word,lemma,stem,tag) = line.split('\t')[:5] if sentnr != lastsentnr: if len(sent) > 0: data.append((sentnr,sent)) sent = [] lastsentnr = sentnr sent.append((word,lemma,tag)) return data import codecs datafile = codecs.open(r"labeledmorph_dutch.csv", "r","utf-8") morphdata = [line for line in datafile if not line.startswith('-1')] testdata = load(morphdata) ``` -------------------------------- ### Analyze German Words with HanTa Tagger Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb Analyzes German words using the initialized HanTa tagger, demonstrating its ability to lemmatize and identify parts of speech. It shows variations in output based on the `taglevel` parameter, providing more detailed morphological information at higher levels. ```python print(tagger.analyze('Überlebende')) print(tagger.analyze('überlebende')) print(tagger.analyze('Überlebende',taglevel=3)) print(tagger.analyze('überlebende',taglevel=3)) ``` ```python print(tagger.analyze('Wohnzimmerschränke',taglevel=3)) print(tagger.analyze('Holzfußboden',taglevel=3)) print(tagger.analyze('Schneeballsystem',taglevel=3)) print(tagger.analyze('Exportstandorts',taglevel=3)) ``` ```python print(tagger.analyze('Zoom',taglevel=3)) print(tagger.analyze('Lehrplan',taglevel=3)) print(tagger.analyze('Lehrpläne',taglevel=3)) print(tagger.analyze('Landbewirtschaftung',taglevel=3)) ``` ```python print(tagger.analyze('Teilchen',taglevel=3)) print(tagger.analyze('Holzhäuschen',taglevel=3)) print(tagger.analyze('Bundestagspräsidentin',taglevel=3)) print(tagger.analyze('Lehrerin',taglevel=3)) print(tagger.analyze('Freundin',taglevel=3)) print(tagger.analyze('Kollegin',taglevel=3)) print(tagger.analyze('Lehrerinnen',taglevel=3)) print(tagger.analyze('Freundinnen',taglevel=3)) print(tagger.analyze('Genossinnen',taglevel=3)) print(tagger.analyze('Ärztinnen',taglevel=3)) print(tagger.analyze('FDP-Ehrenvorsitzende',taglevel=3)) ``` -------------------------------- ### POST /analyze Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Analyzes a single word to determine its root, morphological structure, and part-of-speech tag based on the specified tag level. ```APIDOC ## POST /analyze ### Description Analyzes a word to return its linguistic structure. The taglevel parameter controls the granularity of the output. ### Method POST ### Endpoint /analyze ### Parameters #### Query Parameters - **word** (string) - Required - The word to analyze. - **taglevel** (integer) - Required - The depth of analysis (0-3). ### Request Example { "word": "walking", "taglevel": 3 } ### Response #### Success Response (200) - **result** (array) - The morphological breakdown and POS tag. #### Response Example { "result": ["walk", [["walk", "VB"], ["ing", "SUF_ING"]], "VBG"] } ``` -------------------------------- ### POST /tag_word Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Returns the probability distribution of possible part-of-speech tags for a given word. ```APIDOC ## POST /tag_word ### Description Calculates the likelihood of different part-of-speech tags for a specific word. ### Method POST ### Endpoint /tag_word ### Parameters #### Query Parameters - **word** (string) - Required - The word to tag. ### Request Example { "word": "love" } ### Response #### Success Response (200) - **tags** (array) - List of tuples containing the tag and its log-probability. #### Response Example { "tags": [["NN", -8.90], ["VB", -9.94]] } ``` -------------------------------- ### Retrieve Probabilistic POS Tags Source: https://context7.com/wartaal/hanta/llms.txt Retrieves all possible POS tags for a given word with associated log-probability scores. The cutoff parameter allows filtering results by probability. ```python from HanTa import HanoverTagger as ht tagger = ht.HanoverTagger('morphmodel_ger.pgz') # Get all probable POS tags print(tagger.tag_word('Angeln')) # Use cutoff=0 for only the most probable tag print(tagger.tag_word('verdachte', cutoff=0)) # Toggle case sensitivity for German noun detection print(tagger.tag_word('angeln', casesensitive=True)) ``` -------------------------------- ### Analyze Dutch Word Morphology with Varying Tag Levels Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Analyzes the Dutch word 'huishoudhulpje' with different tag levels (0 to 3) to show the granularity of morphological information extracted. Higher tag levels provide more detailed lemma and suffix information. ```python print(tagger_nl.analyze('huishoudhulpje')) print(tagger_nl.analyze('huishoudhulpje',taglevel=0)) print(tagger_nl.analyze('huishoudhulpje',taglevel=1)) print(tagger_nl.analyze('huishoudhulpje',taglevel=2)) print(tagger_nl.analyze('huishoudhulpje',taglevel=3)) ``` -------------------------------- ### Perform Sentence-Level POS Tagging Source: https://github.com/wartaal/hanta/blob/master/README.md Uses NLTK tokenization combined with the HanTa tagger to process a full sentence and retrieve POS tags and lemmata for each token. ```python import nltk from pprint import pprint sent = "Die Europawahl in den Niederlanden findet immer donnerstags statt." words = nltk.word_tokenize(sent) lemmata = tagger_de.tag_sent(words) pprint(lemmata) ``` -------------------------------- ### Evaluate Lemmatisation Accuracy (Python) Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb This function evaluates the accuracy of a lemmatiser. It compares the predicted lemmas with the actual lemmas for each word in the sentences. The input is a list of sentences with their true lemmas, and the output is a floating-point number representing the lemmatisation accuracy. ```python def lemma_evaluate(sents): correct = 0 nr = 0 for snr,sent in sents: ws = [w for (w,l,c) in sent] ls = [l for (w,l,c) in sent] pred_ls = [l for _,l,_ in tagger.tag_sent(ws,taglevel = 1)] for i in range(len(ws)): nr += 1 if ls[i] == pred_ls[i]: correct += 1 #elif ls[i].lower() != pred_ls[i].lower(): # print(snr,'\t',' '.join(ws)) # print(ws[i],ls[i],pred_ls[i]) # print() if nr%50 == 0: print(correct/nr,end='\r') return correct/nr ``` -------------------------------- ### Evaluate POS tagging and lemmatization accuracy Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Functions to calculate the accuracy of the tagger by comparing predicted POS tags and lemmas against ground truth data from a test set. ```python def tag_evaluate(sents): correct = 0 nr = 0 for snr,sent in sents: ws = [w for (w,l,c) in sent] cs = [c for (w,l,c) in sent] pred_cs = tagger.tag_sent(ws,taglevel = 0) for i in range(len(ws)): nr += 1 if cs[i] == pred_cs[i]: correct += 1 return correct/nr def lemma_evaluate(sents): correct = 0 nr = 0 for snr,sent in sents: ws = [w for (w,l,c) in sent] ls = [l for (w,l,c) in sent] pred_ls = [l for _,l,_ in tagger.tag_sent(ws,taglevel = 1)] for i in range(len(ws)): nr += 1 if ls[i] == pred_ls[i]: correct += 1 return correct/nr ``` -------------------------------- ### Tag words for part-of-speech probability Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Retrieves potential part-of-speech tags for a given word along with their associated probability scores. ```python print(tagger.tag_word('love')) print(tagger.tag_word('school')) print(tagger.tag_word('eating')) tagger.tag_word('intimidating') ``` -------------------------------- ### POST /tag_sent Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb Processes a list of tokens representing a sentence and returns POS tags for each word based on context and tag level. ```APIDOC ## POST /tag_sent ### Description Performs part-of-speech tagging on a full sentence provided as a list of tokens. ### Method POST ### Endpoint /tag_sent ### Parameters #### Request Body - **tokens** (array) - Required - List of words in the sentence. - **taglevel** (integer) - Optional - Depth of tagging (0-3). ### Request Example { "tokens": ["Tackling", "the", "entire", "kitchen"], "taglevel": 1 } ### Response #### Success Response (200) - **tagged_sentence** (array) - List of tuples containing word, root, and tag. #### Response Example { "tagged_sentence": [["Tackling", "tackl", "VBG"], ["the", "the", "AT"]] } ``` -------------------------------- ### Tag a word Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Retrieves the most probable POS-tags for a word, including log-probability scores. ```APIDOC ## POST /tag_word ### Description Returns a list of possible POS-tags for a word with their associated log-probabilities. ### Method POST ### Endpoint /tag_word ### Parameters #### Request Body - **word** (string) - Required - The word to tag. - **cutoff** (float) - Optional - Max difference in logprob from the best result. - **casesensitive** (boolean) - Optional - Whether to respect case for POS guessing. ### Request Example { "word": "Angeln", "cutoff": 10, "casesensitive": true } ### Response #### Success Response (200) - **tags** (array) - List of tuples containing tag and log-probability. #### Response Example { "tags": [["NN", -13.007], ["NNI", -13.704]] } ``` -------------------------------- ### Evaluate POS Tagging Accuracy (Python) Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb This function evaluates the accuracy of a Part-of-Speech (POS) tagger. It iterates through sentences, compares the predicted tags with the actual tags, and calculates the overall accuracy. The input is a list of sentences with their true tags, and the output is a floating-point number representing the accuracy. ```python def tag_evaluate(sents): correct = 0 nr = 0 for snr,sent in sents: ws = [w for (w,l,c) in sent] cs = [c for (w,l,c) in sent] pred_cs = tagger.tag_sent(ws,taglevel = 0) for i in range(len(ws)): nr += 1 if cs[i] == pred_cs[i]: correct += 1 #else: # c = cs[i].split('(')[0] # pred_c = pred_cs[i].split('(')[0] # if c != pred_c: # print(snr,'\t',' '.join(ws)) # print(ws[i],cs[i],pred_cs[i]) # print() if nr%50 == 0: print(correct/nr,end='\r') return correct/nr ``` -------------------------------- ### Tag individual words Source: https://github.com/wartaal/hanta/blob/master/nl/TestDutch.ipynb Uses tagger.tag_word to retrieve potential part-of-speech tags and their associated probability scores for a given word. ```python tagger.tag_word('voorspellingsmogelijkheden') tagger.tag_word('bodemvochtigheid') tagger.tag_word('ware') ``` -------------------------------- ### Tag Single English Word with Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Tags the English word 'walks' and returns a list of possible part-of-speech tags (e.g., VBZ, NNS) with their confidence scores. ```python tagger_en.tag_word('walks') ``` -------------------------------- ### Analyze a word Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Analyzes a single word to retrieve its lemma, part-of-speech tag, and morphological structure based on a specified tag level. ```APIDOC ## POST /analyze ### Description Returns the most probable part of speech, lemma, and morphological analysis for a given word. ### Method POST ### Endpoint /analyze ### Parameters #### Request Body - **word** (string) - Required - The word to analyze. - **taglevel** (integer) - Optional - Amount of detail (0-3). - **pos** (string) - Optional - Force a specific part-of-speech tag. ### Request Example { "word": "Fachmärkte", "taglevel": 3 } ### Response #### Success Response (200) - **result** (array) - The morphological analysis and POS tag. #### Response Example { "result": ["fachmarkt", [["fach", "NN"], ["märkt", "NN_VAR"], ["e", "SUF_NN"]], "NN"] } ``` -------------------------------- ### POST /tag_sent Source: https://context7.com/wartaal/hanta/llms.txt Tags all words in a sentence using trigram-based contextual disambiguation. The taglevel parameter controls the granularity of the returned information. ```APIDOC ## POST /tag_sent ### Description Tags all words in a sentence using trigram-based contextual disambiguation. The method computes probabilities for each word and selects the contextually most appropriate POS tag. ### Method POST ### Endpoint /tag_sent ### Parameters #### Query Parameters - **sent** (list) - Required - A list of tokens representing the sentence. - **taglevel** (int) - Optional - Controls output detail: 0 (POS tags only), 1 (word, lemma, POS), 3 (full morphological analysis). - **casesensitive** (bool) - Optional - Whether the tagging should be case-sensitive. Default is True. ### Request Example { "sent": ["Die", "Europawahl", "in", "den", "Niederlanden", "findet", "."], "taglevel": 1 } ### Response #### Success Response (200) - **result** (list) - A list of tuples containing the word, lemma, and POS tag (depending on taglevel). #### Response Example [ ["Die", "der", "ART"], ["Europawahl", "Europawahl", "NN"] ] ``` -------------------------------- ### Tag German Sentences with HanTa Source: https://context7.com/wartaal/hanta/llms.txt Tags words in a German sentence using trigram-based contextual disambiguation. It supports different tag levels for output detail, from POS tags only to full morphological analysis. Requires NLTK for word tokenization. ```python from HanTa import HanoverTagger as ht import nltk tagge r = ht.HanoverTagger('morphmodel_ger.pgz') sent = "Die Europawahl in den Niederlanden findet immer donnerstags statt." words = nltk.word_tokenize(sent) # taglevel=0: Returns list of POS tags only (fastest) tags = tagger.tag_sent(words, taglevel=0) print(tags) # taglevel=1 (default): Returns list of (word, lemma, POS) lemmata = tagger.tag_sent(words, taglevel=1) print(lemmata) # taglevel=3: Returns full morphological analysis sent2 = "Die Sozialdemokraten haben ersten Prognosen zufolge die Europawahl gewonnen." words2 = nltk.word_tokenize(sent2) analysis = tagger.tag_sent(words2, taglevel=3) # Each element: (word, stem, morpheme_list, POS) print(analysis[1]) # Sozialdemokraten print(analysis[-2]) # gewonnen ``` -------------------------------- ### Tag Single Dutch Word with Hanta Tagger Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Tags a single Dutch word ('staat') and returns a list of possible morphological tags along with their confidence scores. This is useful for disambiguation. ```python tagger_nl.tag_word('staat') ``` -------------------------------- ### Evaluate Lemmatisation Accuracy Source: https://github.com/wartaal/hanta/blob/master/en/TestEnglish.ipynb This Python function evaluates the accuracy of a lemmatiser. It processes sentences, extracting words and their correct lemmas. It then compares these correct lemmas against the lemmas predicted by the 'tagger.tag_sent' method (with taglevel=1). The function calculates and prints the lemmatisation accuracy. ```python def lemma_evaluate(sents): correct = 0 nr = 0 for snr,sent in sents: ws = [w for (w,l,c) in sent] ls = [l for (w,l,c) in sent] pred_ls = [l for _,l,_ in tagger.tag_sent(ws,taglevel = 1)] for i in range(len(ws)): nr += 1 if ls[i].lower() == pred_ls[i].lower(): correct += 1 #else: # print(snr,'\t',' '.join(ws)) # print(ws[i],ls[i],pred_ls[i]) # print() if nr%50 == 0: print(correct/nr,end='\r') return correct/nr ``` -------------------------------- ### Tag Dutch Word for Part-of-Speech and Lemma Source: https://github.com/wartaal/hanta/blob/master/Demo.ipynb Tags the Dutch word 'vertrouwen' and returns a list of potential part-of-speech tags and lemmas with associated scores. This helps in understanding the word's grammatical roles. ```python tagger_nl.tag_word('vertrouwen') ``` -------------------------------- ### Tag German Words with HanTa Tagger Source: https://github.com/wartaal/hanta/blob/master/de/TestGerman.ipynb Tags individual German words using the HanTa tagger to determine their potential parts of speech and associated probabilities. This function is useful for disambiguation and understanding word roles in sentences. ```python print(tagger.tag_word('verdachten')) print(tagger.tag_word('verfahren')) print(tagger.tag_word('Verfahren')) ``` ```python print(tagger.tag_word('verdachten',cutoff=10)) ``` ```python tagger.tag_word('Freundin') ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.