### Load GlotLID Model

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

Download and load the GlotLID model using fastText and huggingface_hub. Ensure you have the necessary libraries installed. The cache_dir can be specified to control model storage location.

```Python
# pip install fasttext-numpy2-wheel  # For newer Python versions ≥ 3.10, install this instead of "fasttext"
# pip install huggingface_hub

import fasttext
from huggingface_hub import hf_hub_download

# download model and get the model path
# cache_dir is the path to the folder where the downloaded model will be stored/cached.
model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin", cache_dir=None)
print("model path:", model_path)

# load the model
model = fasttext.load_model(model_path)
```

--------------------------------

### Get Sentence Vectors

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

Generate a vector representation (embedding) for an input sentence. This can be used for various NLP tasks requiring semantic understanding.

```Python
embedding = model.get_sentence_vector(sent)
```

--------------------------------

### Predict Language Identification (Top K)

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

Get the top k most likely language predictions for a given text, along with their confidence scores. Useful for scenarios requiring multiple possibilities.

```Python
model.predict("Hello, world!", 3)
# (('__label__eng_Latn', '__label__sun_Latn', '__label__ind_Latn'), array([9.98502016e-01, 7.95848144e-04, 3.11827025e-04]))
```

--------------------------------

### Loading Model and Predicting with Limited Languages

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

This Python code demonstrates how to download a GlotLID model from Hugging Face Hub, instantiate the `CustomLID` class with a specific list of languages, and then use it to predict the language of a given text.

```Python
from huggingface_hub import hf_hub_download

# download model and get the model path
# cache_dir is the path to the folder where the downloaded model will be stored/cached.
model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin", cache_dir=None)
print("model path:", model_path)


# to make sure these languages are available in GlotLID check the list of supported labels in model.labels
limited_languages = ['__label__eng_Latn', '__label__arb_Arab', '__label__rus_Cyrl', '__label__por_Latn', '__label__pol_Latn', '__label__hin_Deva']

model = CustomLID(model_path, languages = limited_languages , mode='before')

model.predict("Hello, world!", 3)

```

--------------------------------

### Access Supported Labels

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

Retrieve a list of all language-script labels supported by the loaded GlotLID model. This is useful for understanding the model's capabilities.

```Python
model.labels
# ['__label__eng_Latn', '__label__rus_Cyrl', '__label__arb_Arab', '__label__por_Latn', ...]
```

--------------------------------

### CustomLID Class for Language Limiting

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

This Python class, `CustomLID`, allows you to load a FastText model and limit its predictions to a specified set of languages. It supports two modes: 'before' (default) and 'after' the softmax calculation.

```Python
import fasttext
import numpy as np
class CustomLID:
    def __init__(self, model_path, languages = -1, mode='before'):
        self.model = fasttext.load_model(model_path)
        self.output_matrix = self.model.get_output_matrix()
        self.labels = self.model.get_labels()
        
        # compute language_indices
        if languages !=-1 and isinstance(languages, list):
            self.language_indices = [self.labels.index(l) for l in list(set(languages)) if l in self.labels]

        else:
            self.language_indices = list(range(len(self.labels)))

        # limit labels to language_indices
        self.labels = list(np.array(self.labels)[self.language_indices])
        
        # predict
        self.predict = self.predict_limit_after_softmax if mode=='after' else self.predict_limit_before_softmax

    
    def predict_limit_before_softmax(self, text, k=1):
        
        # sentence vector
        sentence_vector = self.model.get_sentence_vector(text)
        
        # dot
        result_vector = np.dot(self.output_matrix[self.language_indices, :], sentence_vector)

        # softmax
        softmax_result = np.exp(result_vector - np.max(result_vector)) / np.sum(np.exp(result_vector - np.max(result_vector)))

        # top k predictions
        top_k_indices = np.argsort(softmax_result)[-k:][::-1]
        top_k_labels = [self.labels[i] for i in top_k_indices]
        top_k_probs = softmax_result[top_k_indices]

        return tuple(top_k_labels), top_k_probs


    def predict_limit_after_softmax(self, text, k=1):
        
        # sentence vector
        sentence_vector = self.model.get_sentence_vector(text)
        
        # dot
        result_vector = np.dot(self.output_matrix, sentence_vector)

        # softmax
        softmax_result = np.exp(result_vector - np.max(result_vector)) / np.sum(np.exp(result_vector - np.max(result_vector)))

        # limit softmax to language_indices
        softmax_result = softmax_result[self.language_indices]

        
        # top k predictions
        top_k_indices = np.argsort(softmax_result)[-k:][::-1]
        top_k_labels = [self.labels[i] for i in top_k_indices]
        top_k_probs = softmax_result[top_k_indices]

        return tuple(top_k_labels), top_k_probs

```

--------------------------------

### Predict Language Identification (Single)

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

Perform single language identification for a given text. The output includes the predicted label and its confidence score.

```Python
model.predict("Hello, world!")
# (('__label__eng_Latn',), array([0.99850202]))
```

--------------------------------

### GlotLID Citation Information

Source: https://github.com/cisnlp/glotlid/blob/main/README.md

BibTeX entry for citing the GlotLID model in research publications. Use this when referencing the model's contribution to your work.

```bibtex
@inproceedings{
    kargaran2023glotlid,
    title={{G}lot{LID}: Language Identification for Low-Resource Languages},
    author={Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{"u}tze, Hinrich},
    booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
    year={2023},
    url={https://openreview.net/forum?id=dl4e3EBz5j}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.