### Install gRPC Libraries for Python Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/grpc/README.md Commands to install the necessary gRPC libraries for Python client development. ```bash pip install grpcio-tools pip install googleapis-common-protos ``` -------------------------------- ### Start Zemberek gRPC Server with Data Root Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/grpc/README.md Command to start the Zemberek gRPC server, specifying a data root directory for necessary model files. Replace '/home/aaa/zemberek-data' with the actual path. ```bash java -jar zemberek-full.jar StartGrpcServer --dataRoot /home/aaa/zemberek-data ``` -------------------------------- ### Stemming and Lemmatization Example Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/morphology/README.md Finds morphological analyses, stems, and lemmas for the word 'kitabımızsa'. Iterates through each analysis to print details. ```Java TurkishMorphology morphology = TurkishMorphology.createWithDefaults(); WordAnalysis result = morphology.analyze("kitabımızsa"); for (SingleAnalysis analysis : result) { System.out.println(analysis.formatLong()); System.out.println("\tStems = " + analysis.getStems()); System.out.println("\tLemmas = " + analysis.getLemmas()); } ``` -------------------------------- ### Python gRPC Client Example for Language Detection Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/grpc/README.md An example Python script demonstrating how to connect to the Zemberek gRPC server and use the LanguageIdService to detect the language of a given input string. ```python #!/usr/bin/env python3 # -*- coding: utf-8 -*- import grpc import language_id_pb2 as z_langid import language_id_pb2_grpc as z_langid_g channel = grpc.insecure_channel('localhost:6789') langid_stub = z_langid_g.LanguageIdServiceStub(channel) def find_lang_id(i): response = langid_stub.Detect(z_langid.LanguageIdRequest(input=i)) return response.langId def run(): lang_detect_input = 'merhaba dünya' lang_id = find_lang_id(lang_detect_input) print("Language of %s is: %s" % (lang_detect_input, lang_id)) if __name__ == '__main__': run() ``` -------------------------------- ### OpenNLP Style Annotation Example Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/ner/README.md Example of annotating named entities using OpenNLP style with START and END tags. Entity types are specified within the tags. ```text Enerji Verimliliği Merkezi kurucu başkanı Bülent Yeşilata , Ankara'da bir toplantıya katıldı. ``` -------------------------------- ### Start Zemberek gRPC Server Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/grpc/README.md Command to start the Zemberek gRPC server using the zemberek-full.jar. The default port is 6789. ```bash java -jar zemberek-full.jar StartGrpcServer ``` -------------------------------- ### Preprocessed Training Data Example Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/classification/README.md After preprocessing steps like tokenization and lowercasing, the training data format remains the same but with normalized text. ```text __label__magazin jackie chan'a yapmadıklarını bırakmadılar __label__spor fenerbahçe akhisar'da çok rahat kazandı __label__teknoloji google nexus telefonları huawei de üretebilir ``` -------------------------------- ### Bracket Style Annotation Example Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/ner/README.md Example of annotating named entities using bracket style. PER, ORG, and LOC are common types but can be customized. ```text [ORG Enerji Verimliliği Merkezi] kurucu başkanı [PER Bülent Yeşilata] , [LOC Ankara'da] bir toplantıya katıldı. ``` -------------------------------- ### Informal Turkish Words Analysis Setup Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/morphology/README.md Initializes TurkishMorphology to enable informal word analysis. Use this configuration for analyzing non-standard Turkish words. ```Java TurkishMorphology morphology = TurkishMorphology.builder() .setLexicon(RootLexicon.DEFAULT) .useInformalAnalysis() .build(); ``` -------------------------------- ### Morphological Analysis Example Source: https://github.com/ahmetaa/zemberek-nlp/wiki/FAQ Demonstrates the morphological analysis of the Turkish word 'kalemlerimden' to identify its root word and suffixes. ```text [kalem:Noun] kalem:Noun+ler:A3pl+im:P1sg+den:Abl [kalem:Noun] → lemma and root POS. kalem:Noun → Stem. This may be different than lemma. ler:A3pl → Plural suffix `A3pl` with form `ler` im:P1sg → First person singular possessive suffix `P1sg` with form `im` den:Abl → Ablative suffix `Abl` (from) with form "den" ``` -------------------------------- ### Initialize TurkishSentenceNormalizer Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/normalization/README.md Demonstrates how to initialize the TurkishSentenceNormalizer with lookup roots and a language model file. Ensure the paths to the normalization lookup directory and the language model file are correctly set. ```java Path lookupRoot = Paths.get("/home/aaa/zemberek-data/normalization") Path lmFile = Paths.get("/home/aaa/zemberek-data/lm/lm.2gram.slm") TurkishMorphology morphology = TurkishMorphology.createWithDefaults(); TurkishSentenceNormalizer normalizer = new TurkishSentenceNormalizer(morphology, lookupRoot, lmFile); ``` -------------------------------- ### Enamex Style Annotation Example Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/ner/README.md Example of annotating named entities using Enamex style with b_enamex and e_enamex tags. The TYPE attribute specifies the entity type. ```text Enerji Verimliliği Merkezi kurucu başkanı Bülent Yeşilata , Ankara'da bir toplantıya katıldı. ``` -------------------------------- ### Generating Words Programmatically Source: https://github.com/ahmetaa/zemberek-nlp/wiki/FAQ Java code example for programmatically generating words using Zemberek-NLP. ```java [Yes.](https://github.com/ahmetaa/zemberek-nlp/blob/master/examples/src/main/java/zemberek/examples/morphology/GenerateWords.java) ``` -------------------------------- ### Creating TurkishMorphology with Custom Dictionaries Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/morphology/README.md Build a custom RootLexicon by adding default dictionaries and custom text files, then use it to create a TurkishMorphology analyzer. This allows for specialized analysis based on your own word lists. ```java RootLexicon lexicon = RootLexicon.builder() .addDefaultLexicon() .addTextDictionaries(Paths.get("my-dictionary.txt")) .build(); TurkishMorphology analyzer = TurkishMorphology.builder() .setLexicon(lexicon) .build(); ``` ```java TurkishMorphology analyzer = TurkishMorphology.create(lexicon); ``` -------------------------------- ### Get Default Sentence Extractor Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/tokenization/README.md Obtain the default singleton instance of TurkishSentenceExtractor for sentence boundary detection. ```java TurkishSentenceExtractor extractor = TurkishSentenceExtractor.DEFAULT; ``` -------------------------------- ### Instantiate SmoothLm with Builder Pattern Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/lm/README.md Programmatically create a SmoothLm instance using its Builder. You can customize parameters like logBase and unigramSmoothing during instantiation. ```java SmothLm lm = SmoothLm.builder(new File("lm.smooth")).build(); ``` ```java SmoothLm lm = SmoothLm.builder(new File("lm.smooth")) .logBase(Math.E) .unigramSmoothing(0.8) .build(); ``` -------------------------------- ### Compress Language Model from ARPA File Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/lm/README.md Use this command-line application to convert a standard ARPA formatted language model into a compressed SmoothLm binary file. Adjust memory allocation with -Xmx and compression parameters like -spaceUsage for optimal results. ```bash java -Xmx4G -cp [jar file with dependencies] zemberek.apps.lm.CompressLm -in lm.arpa -out lm.smooth ``` -------------------------------- ### Generate LibreOffice Specific JAR Source: https://github.com/ahmetaa/zemberek-nlp/wiki/Zemberek-For-Developers To create a reduced JAR file for LibreOffice integration, first run `mvn install` in the project root, then navigate to the `all` directory and run `mvn -f zemberek-lo.pom.xml package`. ```bash mvn install ``` ```bash mvn -f zemberek-lo.pom.xml package ``` -------------------------------- ### Adding a New Dictionary Item Programmatically Source: https://github.com/ahmetaa/zemberek-nlp/wiki/FAQ Java code example for programmatically adding a new dictionary item to Zemberek-NLP. ```java [Yes.](https://github.com/ahmetaa/zemberek-nlp/blob/master/examples/src/main/java/zemberek/examples/morphology/AddNewDictionaryItem.java) ``` -------------------------------- ### Training a Quantized Classification Model Source: https://github.com/ahmetaa/zemberek-nlp/blob/master/classification/README.md Generate a quantized model for reduced size by using the --applyQuantization and --cutOff flags during training. ```bash java -jar zemberek-full.jar TrainClassifier \ -i news-title-set \ -o news-title.model \ --learningRate 0.1 \ --epochCount 50 \ --applyQuantization \ --cutOff 15000 ```