### Build BNLP Toolkit from Source Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Clone the BNLP repository and install it from source if you need to build it yourself. ```bash git clone https://github.com/sagorbrur/bnlp.git cd bnlp python setup.py install ``` -------------------------------- ### Install BNLP Toolkit Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Install the BNLP toolkit using pip. Use the upgrade flag to ensure you have the latest version. ```bash pip install bnlp_toolkit ``` ```bash pip install -U bnlp_toolkit ``` -------------------------------- ### Install BNLP Toolkit Source: https://github.com/sagorbrur/bnlp/blob/main/notebook/bnlp_colab_training.ipynb Installs the BNLP toolkit and its specific version. This is a prerequisite for using the toolkit's functionalities. ```python # !pip install -U bnlp_toolkit !pip install bnlp-toolkit==4.0.0.dev4 ``` -------------------------------- ### Install FastText Library Source: https://github.com/sagorbrur/bnlp/blob/main/notebook/bnlp_colab_training.ipynb Installs the fasttext library using pip. Restart the runtime after installation. ```python !pip install fasttext ``` -------------------------------- ### Install bnlp with Language Detection Support Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Provides the command to install the bnlp toolkit with the necessary dependencies for language detection functionality. This ensures that the FastText model for language identification is available. ```bash # Install with language detection support pip install bnlp_toolkit[langdetect] ``` -------------------------------- ### CLI Help and Options Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Get help for BNLP commands and their options. Output can be formatted as JSON or use different tokenizer types. ```bash # Get help bnlp --help bnlp tokenize --help # Output as JSON bnlp tokenize "আমি বাংলায় গান গাই।" --json # Use different tokenizer bnlp tokenize "আমি বাংলায় গান গাই।" --type nltk # Sentence tokenization bnlp tokenize "আমি বাংলায় গান গাই। তুমি কি গাও?" --type nltk --sentence # Get similar words with custom count bnlp embedding "বাংলা" --similar --topn 5 ``` -------------------------------- ### Project Dependencies for Setup Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md List of dependencies required for the project, specified with version ranges. Ensure these are compatible with your Python environment. ```python install_requires=[ "sentencepiece>=0.2.0", "gensim>=4.3.3", "nltk", "numpy", "scipy>=1.11.0", "sklearn-crfsuite>=0.5.0", "tqdm>=4.66.3", "ftfy>=6.2.0", "emoji>=2.0.0", "requests", "symspellpy>=6.7.0", ], ``` -------------------------------- ### Basic Tokenization Example Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Demonstrates basic tokenization of Bengali text using the BasicTokenizer. The output is a list of tokens. ```python from bnlp import BasicTokenizer tokenizer = BasicTokenizer() raw_text = "আমি বাংলায় গান গাই।" tokens = tokenizer(raw_text) print(tokens) # output: ["আমি", "বাংলায়", "গান", "গাই", "।"] ``` -------------------------------- ### Proposed CLI Tool Examples Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md Illustrates proposed command-line interface commands for BNLP functionalities like tokenization, Named Entity Recognition (NER), and downloading models. ```bash $ bnlp tokenize "আমি বাংলায় গান গাই।" ["আমি", "বাংলায়", "গান", "গাই", "।"] $ bnlp ner "সজীব ওয়াজেদ জয় ঢাকায় থাকেন।" [("সজীব", "B-PER"), ("ওয়াজেদ", "I-PER"), ...] $ bnlp download all ``` -------------------------------- ### Basic Bengali Tokenization Source: https://github.com/sagorbrur/bnlp/blob/main/notebook/bnlp_colab_testing.ipynb Demonstrates basic tokenization of Bengali text using BasicTokenizer. Ensure bnlp_toolkit is installed. ```python from bnlp import BasicTokenizer tokenizer = BasicTokenizer() raw_text = "আমি বাংলায় গান গাই।" tokens = tokenizer(raw_text) print(tokens) ``` -------------------------------- ### Proposed Async Model Loading Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md Illustrates asynchronous model loading to improve user experience with large models. Includes an example with a progress callback. ```python # Proposed async def load_model(model_type: str) -> Model: ... # Or with progress callback model = BengaliWord2Vec( on_progress=lambda pct: print(f"Loading: {pct}%") ) ``` -------------------------------- ### CLI Word Embeddings Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Get similar words for a given Bengali word using the 'bnlp embedding' command with the --similar flag. ```bash # Get word embeddings (similar words) bnlp embedding "বাংলা" --similar ``` -------------------------------- ### Quick Async Model Loading Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Provides a concise one-liner function `load_model_async` to initiate asynchronous model loading. This simplifies the process of starting background model downloads. The loaded model can be accessed via `get_model()`. ```python from bnlp import load_model_async, BengaliWord2Vec # One-liner to start async loading loader = load_model_async(BengaliWord2Vec) # Get model when ready model = loader.get_model() ``` -------------------------------- ### Get Word Vector and Similar Words from Pretrained Bengali GloVe Source: https://github.com/sagorbrur/bnlp/blob/main/docs/README.md Utilize pre-trained Bengali GloVe word vectors to get the vector representation of a word and find the most similar words. The model is downloaded automatically. ```python from bnlp import BengaliGlove bengali_glove = BengaliGlove() # will automatically download pretrained model word = "গ্রাম" vector = bengali_glove.get_word_vector(word) print(vector.shape) similar_words = bengali_glove.get_closest_word(word) print(similar_words) ``` -------------------------------- ### CLI Text Cleaning Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Clean Bengali text using the 'bnlp clean' command. This example removes email addresses. ```bash # Clean text bnlp clean "hello@example.com আমি বাংলায়" --remove-email ``` -------------------------------- ### CLI Model Management Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Download all available models or specific models using the 'bnlp download' command. List available models with 'bnlp list-models'. ```bash # Download models bnlp download all # Download all models bnlp download word2vec # Download specific model # List available models bnlp list-models ``` -------------------------------- ### Pre-built POS Pipeline Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Create a pre-built Part-of-Speech tagging pipeline with optional cleaning. The output is a list of POS tags. ```python from bnlp import create_pos_pipeline # POS pipeline pos_pipeline = create_pos_pipeline(clean=True) tags = pos_pipeline("আমি ভাত খাই।") ``` -------------------------------- ### Create Sample Text File Source: https://github.com/sagorbrur/bnlp/blob/main/notebook/bnlp_colab_training.ipynb Writes Bengali text content into a file named 'sample.txt'. This file is used as input for training NLP models. ```python %%writefile sample.txt চলতি সপ্তাহের শেষে সর্বজনীন পেনশন কর্মসূচির উদ্বোধন হতে যাচ্ছে। প্রধানমন্ত্রীর কার্যালয় থেকে অর্থ মন্ত্রণালয়ে পাঠানো চিঠিতে বলা হয়েছে, প্রধানমন্ত্রী শেখ হাসিনা আগামী ১৭ আগস্ট বৃহস্পতিবার ভার্চ্যুয়াল পদ্ধতিতে এ কর্মসূচির উদ্বোধন করবেন। ওই দিন সকাল ১০টায় ঢাকায় গণভবন থেকে প্রধানমন্ত্রী পেনশন কর্মসূচির উদ্বোধন করবেন। অনুষ্ঠানে সংযুক্ত থাকবে গোপালগঞ্জ, বাগেরহাট, রংপুর জেলা ও সৌদি আরবের বাংলাদেশ দূতাবাস। এর আগে অর্থ বিভাগ সূত্র প্রথম আলোকে জানিয়েছিল, সমাজের বিভিন্ন শ্রেণি-পেশার মানুষের কথা বিবেচনায় নিয়ে সর্বজনীন পেনশন কর্মসূচি চালু করা হচ্ছে। আপাতত চার শ্রেণির জনগোষ্ঠীর জন্য চার ধরনের পেনশন কর্মসূচি চালু করা হচ্ছে। এগুলোর নাম হচ্ছে প্রগতি, সুরক্ষা, সমতা ও প্রবাসী। এর মধ্যে বেসরকারি খাতের চাকরিজীবীদের জন্য ‘প্রগতি’, স্বকর্মে নিয়োজিত ব্যক্তিদের জন্য ‘সুরক্ষা’, প্রবাসী বাংলাদেশিদের জন্য ‘প্রবাসী’ ও দেশের নিম্ন আয়ের জনগোষ্ঠীর জন্য ‘সমতা’ শীর্ষক কর্মসূচি চালু করা হবে। সর্বজনীন পেনশন কর্মসূচি এমনভাবে করা হচ্ছে, যাতে দেশের ১৮ থেকে ৫০ বছর বয়সী সব নাগরিকই এ ব্যবস্থার আওতায় আসতে পারেন এবং ৬০ বছর বয়স থেকে তাঁরা আজীবন পেনশন পাবেন। শুরুর দিকে চিন্তা না থাকলেও পরে ৫০ বছরের বেশি বয়সীদেরও পেনশন কর্মসূচির আওতায় রাখার সুযোগ তৈরি করা হয়েছে। তারা টানা ১০ বছর চাঁদা দেওয়ার পর পেনশন সুবিধা পাবেন। ``` -------------------------------- ### Proposed Transformer Model Integration Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md Shows how to integrate modern transformer models for Bengali NLP tasks. This includes examples for BanglaBERT and BanglaT5. ```python # Proposed from bnlp.transformers import BanglaBERT, BanglaT5 model = BanglaBERT() embeddings = model.encode("আমি বাংলায় গান গাই।") ``` -------------------------------- ### Proposed BNLP Directory Structure Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md A visual representation of the refactored project's directory layout, highlighting new and modified modules. ```text bnlp/ ├── __init__.py # Public API exports ├── _version.py # Version info ├── core/ # NEW: Core abstractions │ ├── protocols.py # Protocol definitions │ ├── pipeline.py # Pipeline API │ └── base.py # Base classes ├── tokenizer/ │ ├── base.py # NEW: TokenizerProtocol │ ├── basic.py │ ├── nltk.py │ └── sentencepiece.py ├── embedding/ │ ├── base.py # NEW: EmbeddingProtocol │ ├── word2vec.py │ ├── fasttext.py │ ├── glove.py │ ├── doc2vec.py │ └── sentence.py # NEW: Sentence embeddings ├── token_classification/ │ ├── base.py # NEW: TaggerProtocol │ ├── pos.py │ ├── ner.py │ └── trainer.py # Renamed for clarity ├── transformers/ # NEW: Modern models │ ├── bert.py │ └── t5.py ├── cleantext/ │ ├── clean.py │ ├── normalize.py # NEW: Normalization │ └── constants.py ├── augmentation/ # NEW │ └── augmenter.py ├── corpus/ │ ├── corpus.py │ └── _stopwords.py ├── utils/ │ ├── config.py │ ├── downloader.py │ ├── utils.py │ └── logging.py # NEW: Centralized logging ├── cli/ # NEW: Command-line interface │ └── main.py └── py.typed # NEW: PEP 561 marker tests/ ├── conftest.py # NEW: Pytest fixtures ├── fixtures/ # NEW: Test data │ ├── sample_texts.py │ └── mock_models.py ├── unit/ # NEW: Unit tests (mocked) │ ├── test_basic_tokenizer.py │ ├── test_cleantext.py │ └── ... └── integration/ # NEW: Integration tests ├── test_word2vec_integration.py └── ... ``` -------------------------------- ### Get Document Vector Source: https://github.com/sagorbrur/bnlp/blob/main/docs/index.rst Obtain the vector representation for a given Bengali document using a pre-trained model. Ensure the model path is correctly specified. ```python from bnlp import BengaliDoc2vec model_key = "NEWS_DOC2VEC" # set this to WIKI_DOC2VEC for model trained on Wikipedis bn_doc2vec = BengaliDoc2vec(model_path=model_key) # if model_path path is not passed NEWS_DOC2VEC will be selected document = "রাষ্ট্রবিরোধী ও উসকানিমূলক বক্তব্য দেওয়ার অভিযোগে গাজীপুরের গাছা থানায় ডিজিটাল নিরাপত্তা আইনে করা মামলায় আলোচিত ‘শিশুবক্তা’ রফিকুল ইসলামের বিরুদ্ধে অভিযোগ গঠন করেছেন আদালত। ফলে মামলার আনুষ্ঠানিক বিচার শুরু হলো। আজ বুধবার (২৬ জানুয়ারি) ঢাকার সাইবার ট্রাইব্যুনালের বিচারক আসসামছ জগলুল হোসেন এ অভিযোগ গঠন করেন। এর আগে, রফিকুল ইসলামকে কারাগার থেকে আদালতে হাজির করা হয়। এরপর তাকে নির্দোষ দাবি করে তার আইনজীবী শোহেল মো. ফজলে রাব্বি অব্যাহতি চেয়ে আবেদন করেন। অন্যদিকে, রাষ্ট্রপক্ষ অভিযোগ গঠনের পক্ষে শুনানি করেন। উভয় পক্ষের শুনানি শেষে আদালত অব্যাহতির আবেদন খারিজ করে অভিযোগ গঠনের মাধ্যমে বিচার শুরুর আদেশ দেন। একইসঙ্গে সাক্ষ্যগ্রহণের জন্য আগামী ২২ ফেব্রুয়ারি দিন ধার্য করেন আদালত।" vector = bn_doc2vec.get_document_vector(text) print(vector.shape) ``` -------------------------------- ### Implement Mock/Fixture Infrastructure for Testing Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md The absence of a 'conftest.py', pytest fixtures, and mocking capabilities hinders effective unit testing. Introduce these elements to facilitate mocking, data fixtures, and parametrized tests. ```Python # No explicit code snippet provided, but the concept implies the need for pytest configuration and mocking utilities. ``` -------------------------------- ### Bengali Word2Vec: Get Word Vector (Custom Model) Source: https://github.com/sagorbrur/bnlp/blob/main/docs/index.rst Generate a word vector using a custom trained BengaliWord2Vec model by specifying the model path. ```python from bnlp import BengaliWord2Vec own_model_path = "own_directory/own_bwv_model.pkl" bwv = BengaliWord2Vec(model_path=own_model_path) word = 'গ্রাম' vector = bwv.get_word_vector(word) print(vector.shape) ``` -------------------------------- ### Pre-built Tokenization Pipeline Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Create a pre-built tokenization pipeline with optional cleaning and specified tokenizer type. The output is a list of tokens. ```python from bnlp import create_tokenization_pipeline # Tokenization pipeline tokenizer_pipeline = create_tokenization_pipeline(clean=True, tokenizer_type="basic") tokens = tokenizer_pipeline("আমি বাংলায় গান গাই।") ``` -------------------------------- ### Bengali Word2Vec: Get Word Vector (Pretrained) Source: https://github.com/sagorbrur/bnlp/blob/main/docs/index.rst Retrieve the vector representation for a Bengali word using a pretrained BengaliWord2Vec model. The model is downloaded automatically. ```python from bnlp import BengaliWord2Vec bwv = BengaliWord2Vec() word = 'গ্রাম' vector = bwv.get_word_vector(word) print(vector.shape) ``` -------------------------------- ### Get word vector using BengaliGloVe Source: https://github.com/sagorbrur/bnlp/blob/main/notebook/bnlp_colab_testing.ipynb Retrieves the vector representation for a given Bengali word using a pre-trained GloVe model. The model is automatically downloaded if not present. ```python from bnlp import BengaliGlove bengali_glove = BengaliGlove() # will automatically download pretrained model word = "গ্রাম" vector = bengali_glove.get_word_vector(word) print(vector.shape) ``` -------------------------------- ### Batch Tokenization, NER Tagging, and Text Cleaning Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Demonstrates how to perform tokenization, Named Entity Recognition (NER) tagging, and text cleaning on multiple texts simultaneously using batch processing utilities. Ensure the necessary tokenizer, NER model, or cleaner is initialized before use. ```python from bnlp import BasicTokenizer, tokenize_batch, tag_batch, clean_batch from bnlp import BengaliNER, CleanText # Batch tokenization tokenizer = BasicTokenizer() texts = ["আমি বাংলায় গান গাই।", "তুমি কোথায় যাও?", "সে বই পড়ে।"] results = tokenize_batch(tokenizer.tokenize, texts) print(results) # Output: [['আমি', 'বাংলায়', ...], ['তুমি', 'কোথায়', ...], ['সে', 'বই', ...]] # Batch NER tagging ner = BengaliNER() texts = ["সজীব ঢাকায় থাকেন।", "রবীন্দ্রনাথ ঠাকুর কলকাতায় জন্মগ্রহণ করেন।"] results = tag_batch(ner.tag, texts) # Batch text cleaning cleaner = CleanText(remove_url=True, remove_email=True) texts = ["email@example.com আমি", "https://example.com তুমি"] results = clean_batch(cleaner, texts) ``` -------------------------------- ### Get Word Vector from Custom Bengali FastText Model Source: https://github.com/sagorbrur/bnlp/blob/main/docs/README.md Generate a word vector using your own trained Bengali FastText model. Specify the path to your model file. ```python from bnlp.embedding.fasttext import BengaliFasttext own_model_path = "own_directory/own_fasttext_model.bin" bft = BengaliFasttext(model_path=own_model_path) word = "গ্রাম" word_vector = bft.get_word_vector(own_model_path, word) print(word_vector.shape) ``` -------------------------------- ### Async Model Loading with Callbacks Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Shows how to load large NLP models asynchronously in the background without blocking the main application thread. It includes callbacks for progress updates (`on_progress`) and completion (`on_complete`). The model is retrieved using `get_model()`, which blocks until loading is finished. ```python from bnlp import AsyncModelLoader, BengaliWord2Vec # Create async loader with callbacks def on_progress(progress): print(f"Loading: {progress.progress * 100:.0f}% - {progress.message}") loader = AsyncModelLoader( BengaliWord2Vec, on_progress=on_progress, on_complete=lambda m: print("Model ready!") ) # Start loading in background loader.start_loading() # Do other work while model loads... print("Doing other work...") # Get model when needed (blocks until ready) model = loader.get_model() vector = model.get_word_vector("বাংলা") ``` -------------------------------- ### Deprecation Warning Strategy Source: https://github.com/sagorbrur/bnlp/blob/main/docs/REFACTORING_ANALYSIS.md Use this pattern to deprecate old methods while guiding users to new ones. Ensure `stacklevel` is set correctly for accurate warning origins. ```python import warnings def old_method(self, ...): warnings.warn( "old_method is deprecated, use new_method instead", DeprecationWarning, stacklevel=2 ) return self.new_method(...) ``` -------------------------------- ### Language Detection Options Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Illustrates various configuration options for the `LanguageDetector`, including setting a custom confidence threshold, specifying a model path, disabling auto-download, and retrieving language names. ```python from bnlp import LanguageDetector # Custom confidence threshold detector = LanguageDetector(threshold=0.7) # Use your own model file detector = LanguageDetector(model_path="/path/to/lid.176.ftz") # Disable auto-download detector = LanguageDetector(auto_download=False) # Get language name print(detector.get_language_name('bn')) # 'Bengali' print(detector.get_language_name('en')) # 'English' ``` -------------------------------- ### Pre-built NER Pipeline Source: https://github.com/sagorbrur/bnlp/blob/main/README.md Create a pre-built Named Entity Recognition pipeline with optional cleaning. The output is a list of recognized entities. ```python from bnlp import create_ner_pipeline # NER pipeline ner_pipeline = create_ner_pipeline(clean=True) entities = ner_pipeline("সজীব ঢাকায় থাকেন।") ``` -------------------------------- ### Pre-train or Resume Bengali Word2Vec Training Source: https://github.com/sagorbrur/bnlp/blob/main/docs/archive/doc_v3.3.2.md Continue training a Word2Vec model or start pre-training with new data. This method allows resuming from a previously trained model. ```python from bnlp import BengaliWord2Vec bwv = BengaliWord2Vec() trained_model_path = "mytrained_model.model" data_file = "raw_text.txt" model_name = "test_model.model" vector_name = "test_vector.vector" bwv.pretrain(trained_model_path, data_file, model_name, vector_name, epochs=5) ```