PWESuite

https://github.com/zouharvi/pwesuite
Admin
PWESuite is an evaluation suite and toolkit for phonetic word embeddings, providing benchmarks for...
Tokens:4,539
Snippets:28
Trust Score:9.7
Update:5 days ago
Show doc for...
Tokens:
Raw
### Development Environment Setup

Source: https://github.com/zouharvi/pwesuite/blob/master/README.md

Commands to clone the repository, create a virtual environment, and install the package in editable mode.

```bash
> git clone git@github.com:zouharvi/pwesuite.git
> cd pwesuite 
> git branch crazy-feature
> python -m venv dev-venv
> source dev-venv/bin/activate
> pip install -e .
```

--------------------------------

### Clone and install PWESuite

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Clones the PWESuite repository, installs it in editable mode with pip, and changes into the repository directory. The output shows that the repository already exists and that all dependencies are already satisfied.

```bash
!git clone https://github.com/zouharvi/pwesuite.git
!pip3 install -e pwesuite
%cd pwesuite
```

```text
fatal: destination path 'pwesuite' already exists and is not an empty directory.
Obtaining file:///home/vilda/pwesuite/pwesuite
  Preparing metadata (setup.py) ... done
Requirement already satisfied: panphon2>=0.3 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (0.3.2)
Requirement already satisfied: numpy>=1.21 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (1.26.4)
Requirement already satisfied: transformers>=4.22 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (4.44.2)
Requirement already satisfied: wandb>=0.13 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (0.16.2)
Requirement already satisfied: panphon>=0.20 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (0.20.0)
Requirement already satisfied: epitran>=1.20 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (1.24)
Requirement already satisfied: tqdm>=4.64 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (4.66.5)
Requirement already satisfied: scikit-learn>=1.1 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (1.4.2)
Requirement already satisfied: scipy>=1.7 in /home/vilda/.local/lib/python3.11/site-packages (from PWESuite==0.0.1) (1.10.1)
Requirement already satisfied: emoji>=2.2 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (2.9.0)
Requirement already satisfied: Levenshtein>=0.20.9 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (0.23.0)
Requirement already satisfied: multiprocess>=0.70 in /home/vilda/miniconda3/lib/python3.11/site-packages (from PWESuite==0.0.1) (0.70.15)
Requirement already satisfied: setuptools in /home/vilda/miniconda3/lib/python3.11/site-packages (from epitran>=1.20->PWESuite==0.0.1) (75.1.0)
Requirement already satisfied: regex in /home/vilda/miniconda3/lib/python3.11/site-packages (from epitran>=1.20->PWESuite==0.0.1) (2023.10.3)
Requirement already satisfied: marisa-trie in /home/vilda/miniconda3/lib/python3.11/site-packages (from epitran>=1.20->PWESuite==0.0.1) (1.1.0)
Requirement already satisfied: requests in /home/vilda/miniconda3/lib/python3.11/site-packages (from epitran>=1.20->PWESuite==0.0.1) (2.32.3)
Requirement already satisfied: rapidfuzz<4.0.0,>=3.1.0 in /home/vilda/miniconda3/lib/python3.11/site-packages (from Levenshtein>=0.20.9->PWESuite==0.0.1) (3.6.1)
Requirement already satisfied: dill>=0.3.7 in /home/vilda/miniconda3/lib/python3.11/site-packages (from multiprocess>=0.70->PWESuite==0.0.1) (0.3.7)
Requirement already satisfied: unicodecsv in /home/vilda/miniconda3/lib/python3.11/site-packages (from panphon>=0.20->PWESuite==0.0.1) (0.14.1)
Requirement already satisfied: PyYAML in /home/vilda/miniconda3/lib/python3.11/site-packages (from panphon>=0.20->PWESuite==0.0.1) (6.0.1)
Requirement already satisfied: editdistance in /home/vilda/miniconda3/lib/python3.11/site-packages (from panphon>=0.20->PWESuite==0.0.1) (0.6.2)
Requirement already satisfied: munkres in /home/vilda/miniconda3/lib/python3.11/site-packages (from panphon>=0.20->PWESuite==0.0.1) (1.1.4)
Requirement already satisfied: joblib>=1.2.0 in /home/vilda/miniconda3/lib/python3.11/site-packages (from scikit-learn>=1.1->PWESuite==0.0.1) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/vilda/.local/lib/python3.11/site-packages (from scikit-learn>=1.1->PWESuite==0.0.1) (3.1.0)
Requirement already satisfied: filelock in /home/vilda/miniconda3/lib/python3.11/site-packages (from transformers>=4.22->PWESuite==0.0.1) (3.16.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /home/vilda/miniconda3/lib/python3.11/site-packages (from transformers>=4.22->PWESuite==0.0.1) (0.24.6)
Requirement already satisfied: packaging>=20.0 in /home/vilda/miniconda3/lib/python3.11/site-packages (from transformers>=4.22->PWESuite==0.0.1) (21.3)
Requirement already satisfied: safetensors>=0.4.1 in /home/vilda/miniconda3/lib/python3.11/site-packages (from transformers>=4.22->PWESuite==0.0.1) (0.4.3)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /home/vilda/miniconda3/lib/python3.11/site-packages (from transformers>=4.22->PWESuite==0.0.1) (0.19.1)
Requirement already satisfied: Click!=8.0.0,>=7.1 in /home/vilda/miniconda3/lib/python3.11/site-packages (from wandb>=0.13->PWESuite==0.0.1) (8.1.7)
Requirement already satisfied: GitPython!=1.0.0,>=1.0.0 in /home/vilda/miniconda3/lib/python3.11/site-packages (from wandb>=0.13->PWESuite==0.0.1) (3.1.41)
Requirement already satisfied: psutil>=5.0.0 in /home/vilda/.local/lib/python3.11/site-packages (from wandb>=0.13->PWESuite==0.0.1) (5.9.6)
```

--------------------------------

### Load and use RNNMetricLearner model

Source: https://github.com/zouharvi/pwesuite/blob/master/README.md

Example of loading the rnn_metric_learning_token_orth model, preprocessing data, and performing inference with batching.

```python
from models.metric_learning.model import RNNMetricLearner
from models.metric_learning.preprocessor import preprocess_dataset_foreign
from main.utils import load_multi_data
import torch
import tqdm
import math

data = load_multi_data(purpose_key="all")
data = preprocess_dataset_foreign(
  [
    {"token_ort": "Hello", "token_ipa": None},
    {"token_ort": "what", "token_ipa": None},
    {"token_ort": "is", "token_ipa": None},
    {"token_ort": "pwesuite", "token_ipa": None},
  ],
  features="token_ort"
)

model = RNNMetricLearner(
    dimension=300,
    feature_size=data[0][0].shape[1],
)
model.load_state_dict(torch.load("computed/models/rnn_metric_learning_token_orth_all.pt"))

# some cheap paralelization
BATCH_SIZE = 32
data_out = []
for i in tqdm.tqdm(range(math.ceil(len(data) / BATCH_SIZE))):
    batch = [f for f, _ in data[i * BATCH_SIZE:(i + 1) * BATCH_SIZE]]
    data_out += list(
        model.forward(batch).detach().cpu().numpy()
    )

assert len(data) == len(data_out)
assert all([len(x) == 300 for x in data_out])
```

--------------------------------

### Get feature names from CountVectorizer

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Retrieves the feature names (character n-grams) learned by the fitted CountVectorizer model.

```python
model.get_feature_names_out()
```

--------------------------------

### Run evaluation scripts

Source: https://github.com/zouharvi/pwesuite/blob/master/README.md

Execute the evaluation suite using the provided CLI scripts after saving embeddings as Pickle or NPZ files.

```bash
./suite_evaluation/eval_all.py --embd your_embd.pkl
```

```bash
./suite_evaluation/eval_{correlations,human_similarity,retrieval,analogy,rhyme,cognate}.py
```

--------------------------------

### Load PWESuite evaluation dataset

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Loads the 'zouharvi/pwesuite-eval' dataset from Hugging Face using the datasets library. The split is set to 'train'. The download progress is shown in the output.

```python
import datasets
data = datasets.load_dataset("zouharvi/pwesuite-eval", split="train")
```

--------------------------------

### Replicate Dataset Script

Source: https://github.com/zouharvi/pwesuite/blob/master/README.md

Execute the provided shell script to replicate the dataset locally on Linux or MacOS.

```bash
./create_dataset/all.sh
```

--------------------------------

### Run PWESuite evaluation on saved embeddings

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Runs the PWESuite evaluation script (eval_all.py) with the saved embedding file 'embd.npy'. The output shows loading of datasets, progress bars, and final scores for human similarity, correlation, retrieval, analogy, rhyme, and cognate tasks, with an overall score of 0.6121.

```bash
!python3 ./suite_evaluation/eval_all.py --embd embd.npy
```

--------------------------------

### Extract character n-gram features with CountVectorizer

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Creates a CountVectorizer that extracts character n-grams (1-2) with a maximum of 256 features from the 'token_ipa' field of the dataset. The fit_transform call produces a sparse matrix of embeddings.

```python
from sklearn.feature_extraction.text import CountVectorizer

model = CountVectorizer(max_features=256, analyzer="char", ngram_range=(1, 2))
embd = model.fit_transform([x["token_ipa"] for x in data])
```

--------------------------------

### Save embedding matrix to .npy file

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Saves the sparse embedding matrix 'embd' as a dense numpy array to the file 'embd.npy' using np.save. This file is later used as input to the evaluation script.

```python
import numpy as np
np.save("embd", embd.toarray())
```

--------------------------------

### Check embedding matrix shape

Source: https://github.com/zouharvi/pwesuite/blob/master/demo.ipynb

Displays the shape of the embedding matrix 'embd'. The output shows (1738496, 256), indicating 1,738,496 rows and 256 columns.

```python
embd.shape
```
Tokens:
Raw