### Install LatinCy Small Model

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Install the small LatinCy model directly from Hugging Face using pip. This command installs the wheel file for the la-core-web-sm model.

```bash
pip install "la-core-web-sm @ https://huggingface.co/latincy/la_core_web_sm/resolve/main/la_core_web_sm-any-py3-none-any.whl"
```

--------------------------------

### Install Latin core model

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/README.md

Install the specific version of the lg model via pip.

```bash
pip install "la-core-web-lg @ https://huggingface.co/latincy/la_core_web_lg/resolve/main/la_core_web_lg-any-py3-none-any.whl"
```

--------------------------------

### Install LatinCy Medium Model

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Install the medium LatinCy model directly from Hugging Face using pip. This command installs the wheel file for the la-core-web-md model.

```bash
pip install "la-core-web-md @ https://huggingface.co/latincy/la_core_web_md/resolve/main/la_core_web_md-any-py3-none-any.whl"
```

--------------------------------

### Install LatinCy Transformer Model

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Install the transformer LatinCy model directly from Hugging Face using pip. This command installs the wheel file for the la-core-web-trf model.

```bash
pip install "la-core-web-trf @ https://huggingface.co/latincy/la_core_web_trf/resolve/main/la_core_web_trf-any-py3-none-any.whl"
```

--------------------------------

### Train Custom Latin Models with Weasel

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Steps to clone the Weasel repository, install dependencies, download training assets, preprocess data, train models (small, lemma, NER), and evaluate/package them.

```bash
# Clone the repository
git clone https://github.com/diyclassics/la_core_web_xx
cd la_core_web_xx

# Install dependencies
pip install -r requirements.txt

# Download training assets (UD treebanks, LASLA corpus)
weasel run assets

# Preprocess and convert data
weasel run preprocess
weasel run convert

# Train the small model
weasel run train_sm
weasel run train-lemma_sm
weasel run train-ner_sm

# Evaluate and package
weasel run evaluate_sm
weasel run assemble_sm
weasel run package_sm

# Run the full workflow for all model sizes
weasel run all
```

--------------------------------

### Configure spaCy Pipeline for Model Training

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Example configuration file (config_sm.cfg) for a small spaCy model, detailing the pipeline components, architecture, and training parameters.

```ini
# configs/config_sm.cfg - Small model configuration

[nlp]
lang = "la"
pipeline = ["senter","tok2vec","tagger","morphologizer","trainable_lemmatizer","parser","lookup_lemmatizer"]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["LOWER","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.tagger]
factory = "tagger"

[components.morphologizer]
factory = "morphologizer"

[components.trainable_lemmatizer]
factory = "trainable_lemmatizer"
top_k = 3

[components.parser]
factory = "parser"

[components.lookup_lemmatizer]
factory = "lookup_lemmatizer"

[training]
max_steps = 20000
eval_frequency = 200

[training.score_weights]
tag_acc = 0.2
pos_acc = 0.25
morph_acc = 0.15
lemma_acc = 0.25
dep_uas = 0.05
dep_las = 0.05

[initialize]
before_init = {"@callbacks":"customize_tokenizer"}
```

--------------------------------

### Load Latin spaCy Model

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Loads the 'la_core_web_sm' spaCy model and prints its version. Ensure the model is installed before running.

```python
model = 'la_core_web_sm'
nlp = spacy.load(model)
print(f'Loaded model: {model} v{nlp.meta["version"]}')
```

--------------------------------

### Load model in spaCy

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/README.md

Load the installed Latin lg model into a spaCy pipeline.

```python
import spacy
nlp = spacy.load("la_core_web_lg")
```

--------------------------------

### Inspect tokens

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Iterates through the tokens in the Doc object, printing each token, its type, and a list of its available attributes. This example breaks after the first token.

```python
# Get tokens from text

for token in doc:
    print(token)
    print(type(token))
    print([item for item in dir(token) if not item.startswith("_")])
    break
```

--------------------------------

### Load spaCy NLP model

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Loads the specified spaCy language model. Ensure the model is installed in your environment.

```python
model = 'la_core_web_trf'
nlp = spacy.load(model)
print(f'Loaded model: {model} v{nlp.meta["version"]}')
```

--------------------------------

### Morphological Analysis with LatinCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Access detailed morphological features for each token in a processed Latin text using the LatinCy model. This example demonstrates accessing features like case, gender, number, tense, mood, and voice.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

text = "Arma uirumque cano, Troiae qui primus ab oris"
doc = nlp(text)
```

--------------------------------

### Define helper function for enumeration

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

A utility function to print items from a list with their corresponding numbers, starting from 1.

```python
# Helper function
def enumerate_print(l):
    for i, x in enumerate(l, 1):
        print(f"{i}: {x}")
```

--------------------------------

### Define Helper Function for Enumerated Printing

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

A utility function to print items from a list with sequential numbering, starting from 1.

```python
def enumerate_print(l):
    for i, x in enumerate(l, 1):
        print(f"{i}: {x}")
```

--------------------------------

### Convert NER Training Data to spaCy Format

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Python function to convert Named Entity Recognition (NER) annotations from JSON format to spaCy's binary DocBin format for model training. Includes an example of the expected JSON input structure.

```python
import spacy
import srsly
from pathlib import Path
from spacy.tokens import DocBin

def convert_ner_data(lang: str, input_path: Path, output_path: Path):
    """Convert NER annotations to spaCy DocBin format."""
    nlp = spacy.load("la_core_web_lg")
    db = DocBin()

    for entry in srsly.read_json(input_path):
        text = entry["text"]
        doc = nlp(text)
        ents = []

        if entry.get("spans"):
            for span_data in entry["spans"]:
                span = doc.char_span(
                    span_data["start"],
                    span_data["end"],
                    label=span_data["label"]
                )
                if span is not None:
                    ents.append(span)

        doc.ents = ents
        db.add(doc)

    db.to_disk(output_path)

# Example NER training data format (JSON)
# [
#   {
#     "text": "Caesar Galliam uicit",
#     "spans": [
#       {"start": 0, "end": 6, "label": "PERSON"},
#       {"start": 7, "end": 14, "label": "LOC"}
#     ]
#   }
# ]

# Convert training data
convert_ner_data("la", Path("ner-train.json"), Path("ner-train.spacy"))
```

--------------------------------

### Initialize spaCy model

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/2_demo_md.ipynb

Load the la_core_web_md model and verify its version.

```python
# Set up spaCy NLP

model = 'la_core_web_md'
nlp = spacy.load(model)
print(f'Loaded model: {model} v{nlp.meta["version"]}')
```

--------------------------------

### Import required libraries

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/2_demo_md.ipynb

Load necessary modules for data manipulation, machine learning, and NLP processing.

```python
# Imports 

import spacy
import pandas as pd
import numpy as np
from sklearn.manifold import TSNE
from spacy import displacy
from pprint import pprint
```

--------------------------------

### Tokenize Latin Text with spaCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Demonstrates basic tokenization of Latin text using the la_core_web_lg model. Shows how words with '-que' suffixes are handled.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

text1 = "Arma virumque cano"
doc1 = nlp(text1)
print([token.text for token in doc1])
# Output: ['Arma', 'uirum', 'que', 'cano']

# Words where "-que" is part of the stem are preserved
text2 = "ubique atque neque quisque"
doc2 = nlp(text2)
print("\nPreserved -que words (not split):")
print([token.text for token in doc2])
# Output: ['ubique', 'atque', 'neque', 'quisque']
```

--------------------------------

### Prepare Sample Latin Text

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Defines a sample Latin text from Ritchie's fables and performs a simple text replacement for 'v' to 'u'.

```python
text = """Haec narrantur a poetis de Perseo. Perseus filius erat Iovis, maximi deorum; avus eius Acrisius appellabatur. Acrisius volebat Perseum nepotem suum necare; nam propter oraculum puerum timebat. Comprehendit igitur Perseum adhuc infantem, et cum matre in arca lignea inclusit. Tum arcam ipsam in mare coniecit. Danae, Persei mater, magnopere territa est; tempestas enim magna mare turbabat. Perseus autem in sinu matris dormiebat."""

text =  text.replace("v","u").replace("V","U")
```

--------------------------------

### Prepare sample Latin text

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Defines a sample Latin text and performs a simple text transformation by replacing 'v' with 'u' and 'V' with 'U'.

```python
# Get sample text; first story from Ritchie's fables

text = """Haec narrantur a poetis de Perseo. Perseus filius erat Iovis, maximi deorum; avus eius Acrisius appellabatur. Acrisius volebat Perseum nepotem suum necare; nam propter oraculum puerum timebat. Comprehendit igitur Perseum adhuc infantem, et cum matre in arca lignea inclusit. Tum arcam ipsam in mare coniecit. Danae, Persei mater, magnopere territa est; tempestas enim magna mare turbabat. Perseus autem in sinu matris dormiebat."""

text =  text.replace("v","u").replace("V","U")
```

--------------------------------

### Display spaCy Pipeline Components

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Prints the names of the components in the loaded spaCy pipeline. This helps understand the processing steps applied to the text.

```python
pprint(nlp.pipe_names)
```

--------------------------------

### Create spaCy Doc Object

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Processes the prepared Latin text using the loaded spaCy model to create a Doc object, which contains linguistic annotations.

```python
doc = nlp(text)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Imports required libraries for spaCy, data manipulation, and visualization.

```python
import spacy
import pandas as pd
import numpy as np
from sklearn.manifold import TSNE
from spacy import displacy
from pprint import pprint
```

--------------------------------

### Load Text Data for Vector Analysis

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/3_demo_lg.ipynb

Reads text from a file, processes it to remove comments and empty lines, and prepares it for NLP analysis. Assumes the file 'ritchies.txt' exists in the same directory.

```python
# Plot proper_noun vectors with TSNE based on Ritchie's fables

with open('ritchies.txt', 'r') as f:
    contents = f.readlines()
    text = " ".join([line.strip() for line in contents if line.strip() and not line.startswith('#')])
    doc = nlp(text)
```

--------------------------------

### Inspect First Token and its Attributes

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Iterates through the tokens in the Doc object, prints the first token, its type, and a list of its available attributes. This is useful for exploring token properties.

```python
for token in doc:
    print(token)
    print(type(token))
    print([item for item in dir(token) if not item.startswith("_")])
    break
```

--------------------------------

### Visualize Dependency Tree in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Generate a visual representation of the dependency parse tree for Latin text using spaCy's displaCy. This is typically used in Jupyter environments.

```python
from spacy import displacy
displacy.render(doc, style="dep", jupyter=True)
```

--------------------------------

### Import necessary libraries

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Imports required libraries for NLP tasks, including spaCy, pandas, numpy, scikit-learn, and matplotlib.

```python
import spacy
import pandas as pd
import numpy as np
from sklearn.manifold import TSNE
import umap
from spacy import displacy
import matplotlib.pyplot as plt
from pprint import pprint
```

--------------------------------

### Handle Enclitic '-que' in Latin Tokenization

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Demonstrates how spaCy's Latin model correctly tokenizes the enclitic '-que' as a separate conjunction while preserving words where '-que' is part of the stem.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

# "-que" is split as enclitic conjunction
text1 = "Arma uirumque cano"
doc1 = nlp(text1)
print("Tokenization with enclitic -que:")
print([token.text for token in doc1])
```

--------------------------------

### Analyze Dependency Parsing in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Examine the syntactic structure of Latin sentences by printing dependency relations between tokens. Requires a loaded spaCy model.

```python
import spacy
from spacy import displacy

nlp = spacy.load("la_core_web_lg")

text = "Tum arcam ipsam in mare coniecit."
doc = nlp(text)

# Print dependency relations
for token in doc:
    print(f"{token.text:12} ←{token.dep_:12}← {token.head.text}")
```

--------------------------------

### Prepare and visualize proper noun vectors

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/2_demo_md.ipynb

Extracts proper noun vectors from a text file, reduces dimensions using TSNE, and plots the results.

```python
# Plot proper_noun vectors with TSNE based on Ritchie's fables

with open('ritchies.txt', 'r') as f:
    contents = f.readlines()
    text = " ".join([line.strip() for line in contents if line.strip() and not line.startswith('#')])
    doc = nlp(text)
```

```python
# Clearer with fewer elements; so only proper_nouns; extract vectors for text

vector_dict = {}

for item in doc:
    if item.tag_ == "proper_noun":
        vector_dict[item.norm_] = item.vector
        
words = list(vector_dict.keys())
vecs = list(vector_dict.values())
```

```python
# Reduce vectors to 2D with TSNE; make dataframe

tsne = TSNE(n_components=2, perplexity=3, random_state=42)
reduced_vecs = tsne.fit_transform(np.asarray(vecs))
df = pd.DataFrame(reduced_vecs, index=words, columns=['x', 'y'])
df['word'] = df.index
```

```python
# Plot TSNE

ax = df.plot(kind='scatter', x='x', y='y', figsize=(15, 15), title="TSNE lat_core_web_md vectors for proper nouns in Ritchie's Fables")

for idx, row in df.iterrows():
    ax.annotate(row['word'], (row['x'], row['y']))
```

--------------------------------

### Visualize Named Entities in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Render named entities within a Latin text using spaCy's displaCy visualizer. This requires the 'ent' style and setting page=True for full HTML output.

```python
from spacy import displacy
html = displacy.render(doc, style="ent", page=True)
```

--------------------------------

### Create DataFrame of Token Attributes

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/2_demo_md.ipynb

Iterate through the first 25 tokens of a document to extract linguistic attributes and store them in a Pandas DataFrame. Ensure you have the 'pandas' library imported as 'pd'.

```python
data = []

for token in doc[:25]:
    data.append(
        [
            token.text,
            token.norm_,
            token.lower_,
            token.lemma_,
            token.pos_,
            token.tag_,
            token.dep_,
            token.has_vector,
            token.morph,
            token.ent_type_,
            token.text in nlp.vocab,
            token.is_oov,
        ]
    )

df = pd.DataFrame(
    data,
    columns=(
        "text",
        "norm",
        "lower",
        "lemma",
        "pos",
        "tag",
        "dep",
        "has_vector",
        "morph",
        "ent_type",
        "in_vocab",
        "is_oov",
    ),
)

df
```

--------------------------------

### Display spaCy Dependency Parse

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Renders the dependency parse of a sentence using displaCy. Ensure the spaCy model is loaded and text is preprocessed.

```python
# Show dependency parse for sample sentence

text = """Tum arcam ipsam in mare coniecit."""
text =  text.replace("v","u").replace("V","U")

sents = nlp(text).sents

for sent in sents:
    print(f'spaCy dependecy parse for "{sent}"')
    displacy.render(sent, style="dep", jupyter=True)
    break
```

--------------------------------

### Extract and print sentences

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Iterates over the sentences detected in the Doc object and prints each sentence using the helper function.

```python
# Get sentences from text

sents = doc.sents

enumerate_print(sents)
```

--------------------------------

### Batch Process Latin Texts with spaCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Efficiently process multiple Latin texts using spaCy's pipe() method for improved performance. Extracts lemma forms of verbs from each document.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

texts = [
    "Gallia est omnis diuisa in partes tres.",
    "Veni, uidi, uici.",
    "Alea iacta est.",
    "Carthago delenda est."
]

# Process texts in batches
docs = list(nlp.pipe(texts, batch_size=50))

# Extract information from each document
for doc in docs:
    verbs = [token.lemma_ for token in doc if token.pos_ == "VERB"]
    print(f"'{doc.text[:30]}...' → verbs: {verbs}")

# Output:
# 'Gallia est omnis diuisa in pa...' → verbs: ['diuido']
# 'Veni, uidi, uici....' → verbs: ['uenio', 'uideo', 'uinco']
# 'Alea iacta est....' → verbs: ['iacio']
# 'Carthago delenda est....' → verbs: ['deleo']
```

--------------------------------

### Load LatinCy Large Model with spaCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Load the large LatinCy model using spaCy's standard interface. This allows access to all NLP pipeline components. The code also prints the model version and the names of the pipeline components.

```python
import spacy

# Load the large model
nlp = spacy.load("la_core_web_lg")

# Check model version and pipeline components
print(f"Model version: {nlp.meta['version']}")
print(f"Pipeline: {nlp.pipe_names}")
```

--------------------------------

### Highlight Noun Chunks with spaCy

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Highlights noun chunks in a document using displaCy's span visualization. Custom colors can be applied to different span types.

```python
# Noun chunks

selection = doc

selection.spans['NP'] = []
for chunk in selection.noun_chunks:
    if len(chunk) > 1:
        selection.spans['NP'].append(chunk)

colors = {'NP': '#85C1E9'}
options = {'spans_key': 'NP', 'colors': colors}
displacy.render(selection, style="span", jupyter=True, options=options)
```

--------------------------------

### Reduce Vectors and Create DataFrame

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/3_demo_lg.ipynb

Reduces high-dimensional word vectors to 2D using TSNE for visualization. The reduced vectors are then organized into a pandas DataFrame with words as index.

```python
# Reduce vectors to 2D with TSNE; make dataframe

tsne = TSNE(n_components=2, perplexity=3, random_state=42)
reduced_vecs = tsne.fit_transform(np.asarray(vecs))
df = pd.DataFrame(reduced_vecs, index=words, columns=['x', 'y'])
df['word'] = df.index
```

--------------------------------

### Load spaCy Latin Model

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/3_demo_lg.ipynb

Loads the 'la_core_web_lg' spaCy model. This model is essential for processing Latin text with spaCy's NLP pipeline.

```python
model = 'la_core_web_lg'
nlp = spacy.load(model)
print(f'Loaded model: {model} v{nlp.meta["version"]}')
```

--------------------------------

### Visualize Noun Chunks in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Render noun chunks identified in Latin text using spaCy's displaCy visualizer with the 'span' style. Custom spans can be defined for visualization.

```python
from spacy import displacy
doc.spans['NP'] = [chunk for chunk in doc.noun_chunks if len(chunk) > 1]
displacy.render(doc, style="span", options={'spans_key': 'NP'})
```

--------------------------------

### Process Latin Text with LatinCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Process Latin text using the loaded LatinCy model to obtain a spaCy Doc object. This includes tokenization, tagging, parsing, and entity recognition. The text is preprocessed to normalize 'v' to 'u' for classical orthography. The code then iterates through the first 10 tokens to display their text, lemma, part-of-speech, tag, dependency relation, and morphological features.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

# Process Latin text (normalize v→u for classical orthography)
text = """Gallia est omnis divisa in partes tres, quarum unam incolunt Belgae,
aliam Aquitani, tertiam qui ipsorum lingua Celtae, nostra Galli appellantur."""
text = text.replace("v", "u").replace("V", "U")

doc = nlp(text)

# Access tokens and their linguistic annotations
for token in doc[:10]:
    print(f"{token.text:15} {token.lemma_:12} {token.pos_:6} {token.tag_:12} {token.dep_:10} {token.morph}")
```

--------------------------------

### Create a pandas DataFrame from spaCy token attributes

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Iterates over the first 25 tokens of a spaCy document to extract various linguistic features and stores them in a DataFrame.

```python
data = []

for token in doc[:25]:
    data.append(
        [
            token.text,
            token.norm_,
            token.lower_,
            token.lemma_,
            token.pos_,
            token.tag_,
            token.dep_,
            token.has_vector,
            token.morph,
            token.ent_type_,
            token.text in nlp.vocab,
            token.is_oov,
        ]
    )

df = pd.DataFrame(
    data,
    columns=[
        "text",
        "norm",
        "lower",
        "lemma",
        "pos",
        "tag",
        "dep",
        "has_vector",
        "morph",
        "ent_type",
        "in_vocab",
        "is_oov",
    ],
)

df
```

--------------------------------

### Visualize POS Vector Projections

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Reduces and plots two distinct sets of vectors to compare their distribution.

```python
# Create UMAP projections
umap_reducer = umap.UMAP(n_components=2)
reduced_cum_adp = umap_reducer.fit_transform(np.asarray(cum_adp))
reduced_cum_conj = umap_reducer.fit_transform(np.asarray(cum_conj))

# Create DataFrames
df_adp = pd.DataFrame(reduced_cum_adp, columns=['x', 'y'])
df_conj = pd.DataFrame(reduced_cum_conj, columns=['x', 'y'])

# Plot UMAP
plt.figure(figsize=(15, 15))
plt.scatter(df_adp['x'], df_adp['y'], c='purple', label="ADP")
plt.scatter(df_conj['x'], df_conj['y'], c='green', label="SCONJ")


plt.title("UMAP lat_core_web_trf vectors for 'cum' as ADP and SCONJ in Ritchie's Fables")
plt.legend()
plt.show()
```

--------------------------------

### Calculate Word and Document Similarity in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Compute semantic similarity between words or documents using word vectors from the 'la_core_web_lg' model. Ensure the model is loaded.

```python
import spacy

# Load model with vectors (md or lg)
nlp = spacy.load("la_core_web_lg")

# Compare word similarity
word1 = nlp("rex")[0]   # king
word2 = nlp("regina")[0]  # queen
word3 = nlp("miles")[0]   # soldier

print(f"rex ↔ regina: {word1.similarity(word2):.3f}")
print(f"rex ↔ miles:  {word1.similarity(word3):.3f}")

# Document-level similarity
doc1 = nlp("Caesar Galliam uicit")
doc2 = nlp("Pompeius Hispaniam subegit")
print(f"Document similarity: {doc1.similarity(doc2):.3f}")
```

--------------------------------

### Perform Named Entity Recognition in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Identify and label named entities (PERSON, LOC, etc.) in Latin text using a pre-trained spaCy model. Ensure the 'la_core_web_lg' model is loaded.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

text = """Iason et Medea e Thessalia expulsi ad urbem Corinthum uenerunt,
cuius urbis Creon quidam regnum tum obtinebat."""

doc = nlp(text)

# Extract named entities
print("Named Entities:")
for ent in doc.ents:
    print(f"  {ent.text:15} → {ent.label_}")
```

--------------------------------

### Display spaCy Named Entities

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Visualizes named entities recognized in a text using displaCy. The text is first processed by the spaCy model.

```python
# Named entities

text = """Iason et Medea e Thessalia expulsi ad urbem Corinthum venerunt, cuius urbis Creon quidam regnum tum obtinebat."""
text =  text.replace("v","u").replace("V","U")

doc = nlp(text)

print(f'spaCy displayed entities for "{sent}"')
displacy.render(doc, style="ent", jupyter=True)
```

--------------------------------

### Display spaCy Named Entities

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Processes text to identify and display named entities using spaCy's displaCy visualizer. Text is preprocessed to replace 'v' with 'u'.

```python
# Named entities

text = """Iason et Medea e Thessalia expulsi ad urbem Corinthum venerunt, cuius urbis Creon quidam regnum tum obtinebat."""
text =  text.replace("v","u").replace("V","U")

doc = nlp(text)

print(f'spaCy displayed entities for "{text}"')
displacy.render(doc, style="ent", jupyter=True)
```

--------------------------------

### Perform Orthographic Normalization in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Normalize Latin text by converting 'v' to 'u' and 'j' to 'i' using the 'norm_' attribute. This is useful for consistent lookups and analysis.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

# Text with 'v' characters
text = "Vivamus, mea Lesbia, atque amemus"
doc = nlp(text)

# Check normalized forms
for token in doc:
    if token.text != token.norm_:
        print(f"{token.text:15} → normalized: {token.norm_}")

# The norm_ attribute is used internally for lookups
print(f"\nLemma lookup uses normalized form:")
for token in doc[:3]:
    print(f"  {token.text} (norm: {token.norm_}) → lemma: {token.lemma_}")
```

--------------------------------

### Reduce Vectors with UMAP

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Transforms high-dimensional vectors into a 2D DataFrame using the UMAP algorithm.

```python
# Reduce vectors to 2D with UMAP; make dataframe
umap_reducer = umap.UMAP(n_components=2)
reduced_vecs = umap_reducer.fit_transform(np.asarray(vecs))
df = pd.DataFrame(reduced_vecs, index=words, columns=['x', 'y'])
df['word'] = df.index
```

--------------------------------

### Plot TSNE Visualization of Word Vectors

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/3_demo_lg.ipynb

Generates a scatter plot of the 2D TSNE-reduced word vectors. Annotates each point with the corresponding word, visualizing the relationships between proper nouns.

```python
# Plot TSNE

ax = df.plot(kind='scatter', x='x', y='y', figsize=(15, 15), title="TSNE lat_core_web_lg vectors for proper nouns in Ritchie's Fables")

for idx, row in df.iterrows():
    ax.annotate(row['word'], (row['x'], row['y']))
```

--------------------------------

### Extract Vectors by POS Tag

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Collects vectors for a specific word based on its part-of-speech tag.

```python
# Extract vectors for "cum" as ADP or SCONJ

cum_adp = []
cum_conj = []

for item in doc:
    if item.text == "cum":
        if item.pos_ == "ADP":
            cum_adp.append(item.vector)
        elif item.pos_ == "SCONJ":
            cum_conj.append(item.vector)
```

--------------------------------

### Extract Morphological Features in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Iterate through tokens in a spaCy Doc object to extract detailed morphological features like case, number, and gender. Requires a loaded Latin model.

```python
for token in doc:
    if token.pos_ not in ["PUNCT"]:
        morph_dict = token.morph.to_dict()
        print(f"{token.text:12} {token.lemma_:10} {token.pos_:6} Case={morph_dict.get('Case', 'N/A'):4} \
              Number={morph_dict.get('Number', 'N/A'):5} Gender={morph_dict.get('Gender', 'N/A')}")
```

--------------------------------

### Sentence Segmentation with LatinCy

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Perform sentence segmentation on Latin text using the loaded LatinCy model. The code iterates through the detected sentences in the spaCy Doc object and prints each sentence.

```python
import spacy

nlp = spacy.load("la_core_web_lg")

text = "Haec narrantur a poetis de Perseo. Perseus filius erat Iouis, maximi deorum;
auus eius Acrisius appellabatur. Acrisius uolebat Perseum nepotem suum necare."

doc = nlp(text)

# Iterate over sentences
for i, sent in enumerate(doc.sents, 1):
    print(f"{i}: {sent.text.strip()}")
```

--------------------------------

### Extract Proper Noun Vectors

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Filters document tokens for proper nouns and stores their vectors in a dictionary.

```python
vector_dict = {}

for item in doc:
    if item.tag_ == "proper_noun":
        vector_dict[item.norm_] = item.vector
        
words = list(vector_dict.keys())
vecs = list(vector_dict.values())
```

--------------------------------

### Extract and Print Sentences

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Extracts sentences from the Doc object using `doc.sents` and prints them using the `enumerate_print` helper function.

```python
sents = doc.sents

enumerate_print(sents)
```

--------------------------------

### Plot Proper Noun Vectors with TSNE

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/1_demo_sm.ipynb

Plots 2D representations of proper noun vectors from Ritchie's Fables using TSNE. Requires pandas, numpy, and scikit-learn. Vectors are extracted from a spaCy processed document.

```python
# Plot proper_noun vectors with TSNE based on Ritchie's fables

with open('ritchies.txt', 'r') as f:
    contents = f.readlines()
    text = " ".join([line.strip() for line in contents if line.strip() and not line.startswith('#')])
    doc = nlp(text)
```

```python
# Clearer with fewer elements; so only proper_nouns; extract vectors for text

vector_dict = {}

for item in doc:
    if item.tag_ == "proper_noun":
        vector_dict[item.norm_] = item.vector
        
words = list(vector_dict.keys())
vecs = list(vector_dict.values())
```

```python
# Reduce vectors to 2D with TSNE; make dataframe

tsne = TSNE(n_components=2, perplexity=3, random_state=42)
reduced_vecs = tsne.fit_transform(np.asarray(vecs))
df = pd.DataFrame(reduced_vecs, index=words, columns=['x', 'y'])
df['word'] = df.index
```

```python
# Plot TSNE

ax = df.plot(kind='scatter', x='x', y='y', figsize=(15, 15), title="TSNE lat_core_web_sm vectors for proper nouns in Ritchie's Fables")

for idx, row in df.iterrows():
    ax.annotate(row['word'], (row['x'], row['y']))
```

--------------------------------

### Custom spaCy Pipeline Component for Transformer Vectors

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Defines a custom spaCy pipeline component to add transformer-based contextual vectors to tokens. This component maps transformer outputs to spaCy token boundaries.

```python
# Get trf vectors
# cf. https://gist.github.com/yeus/a4d7cc6c97485597eb1e0d7fd720b4e3

from spacy.language import Language
from spacy.tokens import Doc

@Language.factory('trf_vectors')
class TrfContextualVectors:
    """
    Spacy pipeline which add transformer vectors to each token based on user hooks.
    https://spacy.io/usage/processing-pipelines#custom-components-user-hooks
    https://github.com/explosion/spaCy/discussions/6511
    """

    def __init__(self, nlp: Language, name: str):
        self.name = name
        Doc.set_extension("trf_token_vecs", default=None)

    def __call__(self, sdoc):
        # inject hooks from this class into the pipeline
        if type(sdoc) == str:
            sdoc = self._nlp(sdoc)

        # pre-calculate all vectors for every token:

        # calculate groups for spacy token boundaries in the trf vectors
        vec_idx_splits = np.cumsum(sdoc._.trf_data.align.lengths)
        # get transformer vectors and reshape them into one large continous tensor
        trf_vecs = sdoc._.trf_data.tensors[0].reshape(-1, 768)
        # calculate mapping groups from spacy tokens to transformer vector indices
        vec_idxs = np.split(sdoc._.trf_data.align.dataXd, vec_idx_splits)

        # take sum of mapped transformer vector indices for spacy vectors
        vecs = np.stack([trf_vecs[idx].sum(0) for idx in vec_idxs[:-1]])
        sdoc._.trf_token_vecs = vecs

        sdoc.user_token_hooks["vector"] = self.vector
        # sdoc.user_span_hooks["vector"] = self.vector
        # sdoc.user_hooks["vector"] = self.vector
        sdoc.user_token_hooks["has_vector"] = self.has_vector
        # sdoc.user_token_hooks["similarity"] = self.similarity
        # sdoc.user_span_hooks["similarity"] = self.similarity
        # sdoc.user_hooks["similarity"] = self.similarity
        return sdoc

    def vector(self, token):
        return token.doc._.trf_token_vecs[token.i]

    def has_vector(self, token):
        return True
```

--------------------------------

### Plot UMAP Scatter Plot

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Visualizes the 2D vector projections as a scatter plot with annotated labels.

```python
# Plot TSNE

ax = df.plot(kind='scatter', x='x', y='y', figsize=(15, 15), title="UMAP lat_core_web_trf vectors for proper nouns in Ritchie's Fables")

for idx, row in df.iterrows():
    ax.annotate(row['word'], (row['x'], row['y']))
```

--------------------------------

### Extract Proper Noun Vectors

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/3_demo_lg.ipynb

Extracts vectors for words tagged as proper nouns from a spaCy document. Stores the normalized word and its corresponding vector in a dictionary.

```python
# Clearer with fewer elements; so only proper_nouns; extract vectors for text

vector_dict = {}

for item in doc:
    if item.tag_ == "proper_noun":
        vector_dict[item.norm_] = item.vector
        
words = list(vector_dict.keys())
vecs = list(vector_dict.values())
```

--------------------------------

### Extract Noun Chunks in Latin

Source: https://context7.com/diyclassics/la_core_web_lg/llms.txt

Identify and extract noun phrases from Latin text using spaCy's noun_chunks iterator. Requires a loaded spaCy model.

```python
import spacy
from spacy import displacy

nlp = spacy.load("la_core_web_lg")

text = """Perseus filius erat Iouis, maximi deorum; auus eius Acrisius appellabatur."""
doc = nlp(text)

# Extract noun chunks
print("Noun Chunks:")
for chunk in doc.noun_chunks:
    print(f"  '{chunk.text}' (root: {chunk.root.text}, root_dep: {chunk.root.dep_})")
```

--------------------------------

### Add Transformer Vector Component to spaCy Pipeline

Source: https://github.com/diyclassics/la_core_web_lg/blob/main/notebooks/4_demo_trf.ipynb

Adds the custom 'trf_vectors' pipeline component to a spaCy language model and processes text to obtain transformer vectors.

```python
# Add pip, re-run nlp

nlp.add_pipe('trf_vectors', name='trf_vectors', last=True)
doc = nlp(text) # NB: will take a bit of time to run
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.