### Install Dependencies and Start Services

Source: https://github.com/natasha/slovnet/blob/master/README.md

Installs required packages and launches Tensorboard and Jupyter Notebook in the background.

```bash
sudo pip3 install -v \
  jupyter \
  tensorboard

mkdir runs
nohup tensorboard \
  --logdir=runs \
  --host=localhost \
  --port=6006 \
  --reload_interval=1 &

nohup jupyter notebook \
  --no-browser \
  --allow-root \
  --ip=localhost \
  --port=8888 \
  --NotebookApp.token='' \
  --NotebookApp.password='' &
```

--------------------------------

### Install Development Environment

Source: https://github.com/natasha/slovnet/blob/master/README.md

Installs project dependencies and installs the package in editable mode.

```bash
pip3 install -r slovnet/requirements/dev.txt -r slovnet/requirements/gpu.txt
pip3 install -e slovnet
```

--------------------------------

### Install SlovNet

Source: https://context7.com/natasha/slovnet/llms.txt

Install the library via pip.

```bash
pip install slovnet
```

--------------------------------

### Example Log Output

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/main.ipynb

This is an example of the log output format, showing training and testing metrics for different epochs.

```text
# [2020-04-23 17:03:43]    0 0.4045 01_train/01_loss
# [2020-04-23 17:03:43]    0 0.8770 01_train/02_uas
# [2020-04-23 17:03:43]    0 0.8595 01_train/03_las
# [2020-04-23 17:03:44]    0 0.2512 02_test/01_loss
# [2020-04-23 17:03:44]    0 0.9231 02_test/02_uas
# [2020-04-23 17:03:44]    0 0.9103 02_test/03_las
# [2020-04-23 17:39:45]    1 0.3287 01_train/01_loss
# [2020-04-23 17:39:45]    1 0.8975 01_train/02_uas
# [2020-04-23 17:39:45]    1 0.8827 01_train/03_las
# [2020-04-23 17:39:45]    1 0.2286 02_test/01_loss
# [2020-04-23 17:39:45]    1 0.9289 02_test/02_uas
# [2020-04-23 17:39:45]    1 0.9172 02_test/03_las
# [2020-04-23 18:15:48]    2 0.3106 01_train/01_loss
# [2020-04-23 18:15:48]    2 0.9025 01_train/02_uas
# [2020-04-23 18:15:48]    2 0.8883 01_train/03_las

# [2020-04-23 18:15:48]    2 0.2158 02_test/01_loss
# [2020-04-23 18:15:48]    2 0.9316 02_test/02_uas
# [2020-04-23 18:15:48]    2 0.9208 02_test/03_las
```

--------------------------------

### Project Setup and Initialization

Source: https://github.com/natasha/slovnet/blob/master/scripts/03_bert_morph/main.ipynb

Initializes the project environment by running the main script, enabling autoreload, and creating necessary directories. It also sets up an S3 client for data management.

```python
%run main.py
%load_ext autoreload
%autoreload 2

!mkdir -p {DATA_DIR} {BERT_DIR} {MODEL_DIR}
s3 = S3()
```

--------------------------------

### Set up Ubuntu Instance for Python Development

Source: https://github.com/natasha/slovnet/blob/master/README.md

Commands to configure locale, update package lists, and install pip on an Ubuntu system. Essential for setting up a Python development environment.

```bash
sudo locale-gen ru_RU.UTF-8

sudo apt-get update
sudo apt-get install -y \
  python3-pip
```

--------------------------------

### Set up Python Virtual Environment for Natasha-Slovnet

Source: https://github.com/natasha/slovnet/blob/master/README.md

Commands to create and activate a Python virtual environment and install project dependencies. Ensure you have Python 3 installed.

```bash
python -m venv ~/.venvs/natasha-slovnet
source ~/.venvs/natasha-slovnet/bin/activate

pip install -r requirements/dev.txt
pip install -e .
```

--------------------------------

### Setup Project Environment

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/main.ipynb

Initializes the project environment by running the main script, loading autoreload, and creating necessary directories. It also initializes an S3 client for data management.

```python
%run main.py
%load_ext autoreload
%autoreload 2

!mkdir -p {DATA_DIR} {NAVEC_DIR} {MODEL_DIR}
s3 = S3()
```

--------------------------------

### Install Slovnet Dependencies

Source: https://github.com/natasha/slovnet/blob/master/scripts/README.md

Run these commands in the repository root to install the necessary development requirements and the package in editable mode.

```bash
pip3 install -r requirements/dev.txt
pip3 install -e .
```

--------------------------------

### Training Logs

Source: https://github.com/natasha/slovnet/blob/master/scripts/04_bert_syntax/main.ipynb

Example output logs showing loss and accuracy metrics during training.

```text
# [2020-04-23 08:34:18]    0 0.4612 01_train/01_loss
# [2020-04-23 08:34:18]    0 0.9047 01_train/02_uas
# [2020-04-23 08:34:18]    0 0.8783 01_train/03_las
# [2020-04-23 08:34:19]    0 0.3214 02_test/01_loss
# [2020-04-23 08:34:19]    0 0.9618 02_test/02_uas
# [2020-04-23 08:34:19]    0 0.9300 02_test/03_las
# [2020-04-23 08:39:07]    1 0.2017 01_train/01_loss
# [2020-04-23 08:39:07]    1 0.9511 01_train/02_uas
# [2020-04-23 08:39:07]    1 0.9348 01_train/03_las

# [2020-04-23 08:39:07]    1 0.3035 02_test/01_loss
# [2020-04-23 08:39:07]    1 0.9635 02_test/02_uas
# [2020-04-23 08:39:07]    1 0.9311 02_test/03_las
```

--------------------------------

### Training Log Output Example

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/main.ipynb

This is an example of the log output generated during the training process, showing metrics like loss and scores for different entities (PER, LOC, ORG) across epochs for both training and testing sets.

```text
# [2020-03-31 14:05:40]    0 14.3334 01_train/01_loss
# [2020-03-31 14:05:43]    0 2.3965 02_test/01_loss
# [2020-03-31 14:05:43]    0 0.9962 02_test/02_PER
# [2020-03-31 14:05:43]    0 0.9807 02_test/03_LOC
# [2020-03-31 14:05:43]    0 0.9691 02_test/04_ORG
# [2020-03-31 14:06:10]    1 1.8448 01_train/01_loss
# [2020-03-31 14:06:13]    1 2.1326 02_test/01_loss
# [2020-03-31 14:06:13]    1 0.9975 02_test/02_PER
# [2020-03-31 14:06:13]    1 0.9862 02_test/03_LOC
# [2020-03-31 14:06:13]    1 0.9710 02_test/04_ORG
# [2020-03-31 14:06:40]    2 1.2753 01_train/01_loss
# [2020-03-31 14:06:43]    2 2.1436 02_test/01_loss
# [2020-03-31 14:06:43]    2 0.9972 02_test/02_PER
# [2020-03-31 14:06:43]    2 0.9867 02_test/03_LOC
# [2020-03-31 14:06:43]    2 0.9705 02_test/04_ORG
# [2020-03-31 14:07:10]    3 1.1283 01_train/01_loss
# [2020-03-31 14:07:13]    3 2.1885 02_test/01_loss
# [2020-03-31 14:07:13]    3 0.9975 02_test/02_PER
# [2020-03-31 14:07:13]    3 0.9867 02_test/03_LOC
# [2020-03-31 14:07:13]    3 0.9719 02_test/04_ORG
# [2020-03-31 14:07:40]    4 1.0464 01_train/01_loss

# [2020-03-31 14:07:43]    4 2.1705 02_test/01_loss
# [2020-03-31 14:07:43]    4 0.9977 02_test/02_PER
# [2020-03-31 14:07:43]    4 0.9862 02_test/03_LOC
# [2020-03-31 14:07:43]    4 0.9722 02_test/04_ORG
```

--------------------------------

### Training Log Output Example

Source: https://github.com/natasha/slovnet/blob/master/scripts/03_bert_morph/main.ipynb

This is an example log output showing training progress, including epoch number, loss, and accuracy for both training and testing sets.

```text
# [2020-03-31 14:26:22]    0 0.3800 01_train/01_loss
# [2020-03-31 14:26:22]    0 0.9177 01_train/02_acc
# [2020-03-31 14:26:22]    0 0.4991 02_test/01_loss
# [2020-03-31 14:26:22]    0 0.9356 02_test/02_acc
# [2020-03-31 14:27:44]    1 0.0919 01_train/01_loss
# [2020-03-31 14:27:44]    1 0.9740 01_train/02_acc
# [2020-03-31 14:27:44]    1 0.5095 02_test/01_loss
# [2020-03-31 14:27:44]    1 0.9388 02_test/02_acc
# [2020-03-31 14:29:06]    2 0.0655 01_train/01_loss
# [2020-03-31 14:29:06]    2 0.9808 01_train/02_acc
# [2020-03-31 14:29:06]    2 0.5176 02_test/01_loss
# [2020-03-31 14:29:06]    2 0.9429 02_test/02_acc
# [2020-03-31 14:30:28]    3 0.0463 01_train/01_loss
# [2020-03-31 14:30:28]    3 0.9861 01_train/02_acc
# [2020-03-31 14:30:28]    3 0.5389 02_test/01_loss
# [2020-03-31 14:30:28]    3 0.9423 02_test/02_acc
# [2020-03-31 14:31:50]    4 0.0344 01_train/01_loss
# [2020-03-31 14:31:50]    4 0.9895 01_train/02_acc

# [2020-03-31 14:31:50]    4 0.5649 02_test/01_loss
# [2020-03-31 14:31:50]    4 0.9416 02_test/02_acc
```

--------------------------------

### Optimizer and Scheduler Setup

Source: https://github.com/natasha/slovnet/blob/master/scripts/03_bert_morph/main.ipynb

Configures the Adam optimizer with different learning rates for the encoder and morphological head, and sets up an ExponentialLR scheduler for learning rate decay.

```python
optimizer = optim.Adam([
    dict(params=encoder.parameters(), lr=BERT_LR),
    dict(params=morph.parameters(), lr=LR),
])
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, LR_GAMMA)
```

--------------------------------

### Run Slovnet Tests

Source: https://github.com/natasha/slovnet/blob/master/README.md

Command to execute the test suite for the Slovnet project. This is typically run after installing dependencies.

```bash
make test
```

--------------------------------

### Initialize Environment and Directories

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/pack.ipynb

Sets up the notebook environment and creates necessary directories for model storage.

```python
%run main.py
%load_ext autoreload
%autoreload 2

!mkdir -p {NAVEC_DIR} {MODEL_DIR} {PACK_DIR}
s3 = S3()
```

--------------------------------

### Initialize Project Environment and Data Directories

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Sets up the project environment by loading extensions, reloading modules, creating necessary directories, and initializing an S3 client for data management.

```python
%run main.py
%load_ext autoreload
%autoreload 2

!mkdir -p {DATA_DIR} {RUBERT_DIR} {MODEL_DIR}
s3 = S3()
```

--------------------------------

### Initialize Environment and S3 Connection

Source: https://github.com/natasha/slovnet/blob/master/scripts/04_bert_syntax/data.ipynb

Sets up the Python environment by loading extensions and initializes an S3 client for cloud storage operations.

```python
%run main.py
%load_ext autoreload
%autoreload 2

!mkdir -p {RAW_DIR}
s3 = S3()
```

--------------------------------

### Initialize Logging Boards

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/main.ipynb

Sets up logging boards for tracking training progress. It uses a MultiBoard to combine TensorBoard and LogBoard, creating separate sections for training and testing metrics.

```python
board = MultiBoard([
    TensorBoard(BOARD_NAME, RUNS_DIR),
    LogBoard()
])
boards = {
    TRAIN: board.section(TRAIN_BOARD),
    TEST: board.section(TEST_BOARD),
}
```

--------------------------------

### Initialize S3 and Load BERT Vocabulary

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/infer.ipynb

Initializes the S3 client and ensures the BERT vocabulary file is available locally by downloading it from S3 if necessary.

```python
%run main.py
%load_ext autoreload
%autoreload 2

s3 = S3()
```

```python
if not exists(BERT_VOCAB):
    s3.download(S3_BERT_VOCAB, BERT_VOCAB)
```

--------------------------------

### Prepare Training and Test Batches

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/main.ipynb

Loads and encodes markup data into batches, then splits them into training and testing sets. Batches are moved to the specified device.

```python
lines = load_gz_lines(NERUS)
lines = log_progress(lines, total=NERUS_TOTAL)
items = parse_jl(lines)
markups = (SyntaxMarkup.from_json(_) for _ in items)

encode = SyntaxTrainEncoder(
    words_vocab, shapes_vocab, rels_vocab,
    batch_size=64,
    sort_size=1000,
)
batches = encode(markups)
batches = [_.to(DEVICE) for _ in batches]

size = 25
batches = {
    TEST: batches[:size],
    TRAIN: batches[size:]
}
```

--------------------------------

### Configure Optimizer

Source: https://github.com/natasha/slovnet/blob/master/scripts/04_bert_syntax/main.ipynb

Sets up the Adam optimizer and learning rate scheduler.

```python
optimizer = optim.Adam([
    dict(params=encoder.parameters(), lr=BERT_LR),
    dict(params=chain(head.parameters(), rel.parameters()), lr=LR),
])
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, LR_GAMMA)
```

--------------------------------

### Construct and Load Syntax Model

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/pack.ipynb

Initialize the syntax model architecture and load pre-trained weights into the components.

```python
navec = Navec.load(NAVEC)

words_vocab = Vocab(navec.vocab.words)
shapes_vocab = Vocab([PAD] + SHAPES)
rels_vocab = Vocab.load(RELS_VOCAB)

word = NavecEmbedding(navec)
shape = Embedding(
    vocab_size=len(shapes_vocab),
    dim=SHAPE_DIM,
    pad_id=shapes_vocab.pad_id
)
emb = SyntaxEmbedding(word, shape)
encoder = SyntaxEncoder(
    input_dim=emb.dim,
    layer_dims=LAYER_DIMS,
    kernel_size=KERNEL_SIZE,
)
head = SyntaxHead(
    input_dim=encoder.dim,
    hidden_dim=encoder.dim // 2,
)
rel = SyntaxRel(
    input_dim=encoder.dim,
    hidden_dim=encoder.dim // 2,
    rel_dim=len(rels_vocab)
)
model = Syntax(emb, encoder, head, rel)
model.eval()

model.emb.shape.load(MODEL_SHAPE)
model.encoder.load(MODEL_ENCODER)
model.head.load(MODEL_HEAD)
model.rel.load(MODEL_REL)
```

--------------------------------

### Release Workflow

Source: https://github.com/natasha/slovnet/blob/master/README.md

Steps to commit version changes, tag the release, and push to the repository.

```bash
# Update setup.py version

git commit -am 'Up version'
git tag v0.6.0

git push
git push --tags

# Github Action builds dist and publishes to PyPi
```

--------------------------------

### Initialize TensorBoard Loggers

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Sets up TensorBoard logging for tracking training and testing metrics. Creates a main board and specific sections for training and testing logs.

```python
board = TensorBoard(BOARD_NAME, RUNS_DIR)
train_board = board.section(TRAIN_BOARD)
test_board = board.section(TEST_BOARD)
```

--------------------------------

### Package Model for Deployment

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/pack.ipynb

Serialize the model, arrays, and vocabularies into a pack file.

```python
with DumpPack(PACK) as pack:
    meta = Meta(ID)
    pack.dump_meta(meta)
    
    pack.dump_model(model)
    pack.dump_arrays(arrays)

    pack.dump_vocab(words_vocab, WORD)
    pack.dump_vocab(shapes_vocab, SHAPE)
    pack.dump_vocab(rels_vocab, REL)
```

--------------------------------

### Create a GPU Compute Instance with Yandex Cloud

Source: https://github.com/natasha/slovnet/blob/master/README.md

CLI commands to provision a GPU-enabled virtual machine on Yandex Cloud. This instance is suitable for deep learning tasks.

```bash
yc compute instance create \
  --name gpu \
  --zone ru-central1-a \
  --network-interface subnet-name=default,nat-ip-version=ipv4 \
  --create-boot-disk image-folder-id=standard-images,image-family=ubuntu-1804-lts-ngc,type=network-ssd,size=20 \
  --cores=8 \
  --memory=96 \
  --gpus=1 \
  --ssh-key ~/.ssh/id_rsa.pub \
  --folder-name default \
  --platform-id gpu-standard-v1 \
  --preemptible
```

--------------------------------

### Load and Run Syntax Parser

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/pack.ipynb

Load a packaged model and perform syntax parsing on input text.

```python
syntax = api.Syntax.load(PACK)
syntax.navec(navec)
words = 'Опубликованы новые данные по заражению коронавирусом в Москве'.split()
markup = syntax(words)
show_syntax_markup(markup)
```

--------------------------------

### Package Model for Deployment

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/pack.ipynb

Saves the model, arrays, and vocabularies into a pack file.

```python
with DumpPack(PACK) as pack:
    meta = Meta(ID)
    pack.dump_meta(meta)
    
    pack.dump_model(model)
    pack.dump_arrays(arrays)

    pack.dump_vocab(words_vocab, WORD)
    pack.dump_vocab(shapes_vocab, SHAPE)
    pack.dump_vocab(tags_vocab, TAG)
```

--------------------------------

### Initialize Vocabularies for Words and Shapes

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/main.ipynb

Creates vocabulary objects for words using Navec and for shapes, including a padding token.

```python
words_vocab = Vocab(navec.vocab.words)
shapes_vocab = Vocab([PAD] + SHAPES)
```

--------------------------------

### Initialize Model

Source: https://github.com/natasha/slovnet/blob/master/scripts/04_bert_syntax/main.ipynb

Constructs the syntax model and freezes BERT embedding parameters.

```python
config = RuBERTConfig()
emb = BERTEmbedding.from_config(config)
encoder = BERTEncoder.from_config(config)
head = BERTSyntaxHead(
    input_dim=config.emb_dim,
    hidden_dim=config.emb_dim // 2,
)
rel = BERTSyntaxRel(
    input_dim=config.emb_dim,
    hidden_dim=config.emb_dim // 2,
    rel_dim=len(rels_vocab)
)
model = BERTSyntax(emb, encoder, head, rel)

for param in emb.parameters():
    param.requires_grad = False

model.emb.load(BERT_EMB)
model.encoder.load(BERT_ENCODER)
model = model.to(DEVICE)

criterion = masked_flatten_cross_entropy
```

--------------------------------

### Download Data Dependencies

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/main.ipynb

Checks if necessary data files (NERUS and NAVEC) exist and downloads them if they are missing. NERUS is downloaded from S3, while NAVEC is downloaded using wget.

```python
if not exists(NERUS):
    s3.download(S3_NERUS, NERUS)
    
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
```

--------------------------------

### Download Required Models

Source: https://context7.com/natasha/slovnet/llms.txt

Download the necessary Navec embeddings and specific SlovNet model packs before initialization.

```bash
# Download Navec embeddings (required for all models)
wget https://storage.yandexcloud.net/natasha-navec/packs/navec_news_v1_1B_250K_300d_100q.tar

# Download NER model
wget https://storage.yandexcloud.net/natasha-slovnet/packs/slovnet_ner_news_v1.tar

# Download Morphology model
wget https://storage.yandexcloud.net/natasha-slovnet/packs/slovnet_morph_news_v1.tar

# Download Syntax model
wget https://storage.yandexcloud.net/natasha-slovnet/packs/slovnet_syntax_news_v1.tar
```

--------------------------------

### Configure Optimizer and Scheduler

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/main.ipynb

Initializes the Adam optimizer for model parameters and an ExponentialLR scheduler to adjust the learning rate during training.

```python
optimizer = optim.Adam(model.parameters(), lr=LR)
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, LR_GAMMA)
```

--------------------------------

### Initialize and Configure RuBERT Model

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Initializes the RuBERT model components (embedding, encoder, MLM head) from configuration, loads pre-trained weights from S3, and moves the model to the appropriate device. Sets the position embedding weights to not require gradients.

```python
config = RuBERTConfig()
emb = BERTEmbedding.from_config()
encoder = BERTEncoder.from_config()
head = BERTMLMHead(config.emb_dim, config.vocab_size)
model = BERTMLM(emb, encoder, head)

 # fix pos emb, train on short seqs
emb.position.weight.requires_grad = False

model.emb.load(RUBERT_EMB)
model.encoder.load(RUBERT_ENCODER)
model.head.load(RUBERT_MLM)
model = model.to(DEVICE)

criterion = masked_flatten_cross_entropy
```

--------------------------------

### Syntax Parsing with SlovNet

Source: https://context7.com/natasha/slovnet/llms.txt

Loads Navec and Syntax models, performs syntax parsing on a list of words, and prints token dependency details. Requires Navec and Syntax models to be downloaded.

```python
from navec import Navec
from slovnet import Syntax

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
syntax = Syntax.load('slovnet_syntax_news_v1.tar')
syntax.navec(navec)

words = ['Мальчик', 'читает', 'книгу']
markup = syntax(words)

# Access token details
for token in markup.tokens:
    print(f'Token ID: {token.id}')
    print(f'  Text: {token.text}')
    print(f'  Head ID: {token.head_id} (0 means root)')
    print(f'  Relation: {token.rel}')
    print()

# Convert to JSON
import json
print(f'JSON: {json.dumps(markup.as_json, ensure_ascii=False, indent=2)}')

```

--------------------------------

### Syntax Visualization with IPyMarkup

Source: https://context7.com/natasha/slovnet/llms.txt

Visualizes dependency parsing results using IPyMarkup. Requires ipymarkup and SlovNet Syntax models.

```python
# Install ipymarkup: pip install ipymarkup
from ipymarkup import show_span_ascii_markup, show_dep_ascii_markup
from navec import Navec
from slovnet import NER, Syntax

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')

# Syntax visualization
syntax = Syntax.load('slovnet_syntax_news_v1.tar')
syntax.navec(navec)

words = ['Президент', 'подписал', 'важный', 'указ']
markup = syntax(words)

# Convert to ipymarkup format
deps = []
for token in markup.tokens:
    source = int(token.head_id) - 1
    target = int(token.id) - 1
    if source >= 0 and source != target:
        deps.append([source, target, token.rel])

show_dep_ascii_markup(words, deps)
```

--------------------------------

### Data Preparation and Splitting

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/main.ipynb

Loads text data from a specified source (either CUSTOM_TEXTS or NE5), parses it into items, and converts each item into a BIO-formatted record. The records are then split into training and testing sets.

```python
records = []
lines = load_gz_lines(CUSTOM_TEXTS) if CUSTOM_TUNING else load_gz_lines(NE5)
items = parse_jl(lines)
items = log_progress(items)

for item in items:
    record = SpanMarkup.from_json(item)
    tokens = list(tokenize(record.text))
    record = record.to_bio(tokens)
    records.append(record)

size = round(len(records) * 0.2)

markups = {
    TEST: records[:size],
    TRAIN: records[size:]
}
```

--------------------------------

### Configure Optimizer, AMP, and Scheduler

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Initializes the Adam optimizer for model parameters, sets up automatic mixed precision (AMP) with a specified optimization level, and configures an exponential learning rate scheduler.

```python
optimizer = optim.Adam(model.parameters(), lr=0.0001)
model, optimizer = amp.initialize(model, optimizer, opt_level=O2)
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, 0.999)
```

--------------------------------

### Initialize and Configure BERTNER Model

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/main.ipynb

Initializes the RuBERT configuration, BERT embedding, BERT encoder, and a NER head. It then constructs the BERTNER model, freezes the embedding layer parameters, loads pre-trained weights for the embedding and encoder, and moves the model to the appropriate device.

```python
config = RuBERTConfig()
emb = BERTEmbedding.from_config(config)
encoder = BERTEncoder.from_config(config)
ner = BERTNERHead(config.emb_dim, len(tags_vocab))
model = BERTNER(emb, encoder, ner)

for param in emb.parameters():
    param.requires_grad = False

model.emb.load(BERT_EMB)
model.encoder.load(BERT_ENCODER)
model = model.to(DEVICE)
```

--------------------------------

### Download Model Artifacts

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/pack.ipynb

Download necessary model weights and vocabularies from S3 if they do not exist locally.

```python
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
    s3.download(S3_MODEL_SHAPE, MODEL_SHAPE)
    s3.download(S3_MODEL_ENCODER, MODEL_ENCODER)
    s3.download(S3_MODEL_HEAD, MODEL_HEAD)
    s3.download(S3_MODEL_REL, MODEL_REL)
    s3.download(S3_RELS_VOCAB, RELS_VOCAB)
```

--------------------------------

### Define and Initialize Syntax Model Components

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/main.ipynb

Initializes various components of the Syntax model, including embeddings, encoder, head, and relation predictor. The model is then moved to the appropriate device.

```python
word = NavecEmbedding(navec)
shape = Embedding(
    vocab_size=len(shapes_vocab),
    dim=SHAPE_DIM,
    pad_id=shapes_vocab.pad_id
)
emb = SyntaxEmbedding(word, shape)
encoder = SyntaxEncoder(
    input_dim=emb.dim,
    layer_dims=LAYER_DIMS,
    kernel_size=KERNEL_SIZE,
)
head = SyntaxHead(
    input_dim=encoder.dim,
    hidden_dim=encoder.dim // 2,
)
rel = SyntaxRel(
    input_dim=encoder.dim,
    hidden_dim=encoder.dim // 2,
    rel_dim=len(rels_vocab)
)
model = Syntax(emb, encoder, head, rel)

model = model.to(DEVICE)

criterion = masked_flatten_cross_entropy
```

--------------------------------

### Build and Run SlovNet Syntax Docker Service

Source: https://context7.com/natasha/slovnet/llms.txt

Builds a Docker image for the SlovNet Syntax service and runs it as a container, exposing port 8080. This is for production deployment.

```bash
# Build and run Syntax service
docker build -t slovnet-syntax -f docker/slovnet-syntax/exec/Dockerfile .
docker run -p 8080:8080 slovnet-syntax

```

--------------------------------

### Slovnet Performance Metrics (Speed, Size, RAM)

Source: https://github.com/natasha/slovnet/blob/master/README.md

Table detailing initialization time, disk size, RAM usage, and sentences processed per second for Slovnet and other parsers. Note GPU usage for BERT models.

```plaintext
slovnet
1.0
27
125
450.0

slovnet_bert
5.0
504
3427
200.0 (gpu)

deepavlov_bert
34.0
1427
8704
75.0 (gpu)

udpipe
6.9
45
242
56.2

spacy
9.0
140
579
41.0

stanza
3.0
591
890
12.0
```

--------------------------------

### Build and Run SlovNet Morphology Docker Service

Source: https://context7.com/natasha/slovnet/llms.txt

Builds a Docker image for the SlovNet Morphology service and runs it as a container, exposing port 8080. This is for production deployment.

```bash
# Build and run Morphology service
docker build -t slovnet-morph -f docker/slovnet-morph/exec/Dockerfile .
docker run -p 8080:8080 slovnet-morph

```

--------------------------------

### Prepare Data Encoders and Batches

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Initializes a BERTMLMTrainEncoder for processing text data into batches suitable for training. Loads lines from test and train files, encodes them, and moves the resulting batches to the specified device.

```python
encode = BERTMLMTrainEncoder(
    vocab,
    seq_len=128,
    batch_size=32,
    shuffle_size=10000
)

lines = load_lines(TEST)
batches = encode(lines)
test_batches = [_.to(DEVICE) for _ in batches]

lines = load_lines(TRAIN)
batches = encode(lines)
train_batches = (_.to(DEVICE) for _ in batches)
```

--------------------------------

### Load Navec and Initialize Model Components

Source: https://github.com/natasha/slovnet/blob/master/scripts/06_morph/pack.ipynb

Loads the Navec model and initializes vocabulary and embedding layers. This includes setting up word, shape, and tag embeddings, followed by the encoder and morphological head.

```python
navec = Navec.load(NAVEC)

words_vocab = Vocab(navec.vocab.words)
shapes_vocab = Vocab([PAD] + SHAPES)
tags_vocab = Vocab.load(TAGS_VOCAB)

word = NavecEmbedding(navec)
shape = Embedding(
    vocab_size=len(shapes_vocab),
    dim=SHAPE_DIM,
    pad_id=shapes_vocab.pad_id
)
emb = TagEmbedding(word, shape)
encoder = TagEncoder(
    input_dim=emb.dim,
    layer_dims=LAYER_DIMS,
    kernel_size=KERNEL_SIZE,
)
morph = MorphHead(encoder.dim, len(tags_vocab))
model = Morph(emb, encoder, morph)
model.eval()

model.emb.shape.load(MODEL_SHAPE)
model.encoder.load(MODEL_ENCODER)
model.head.load(MODEL_MORPH)
```

--------------------------------

### Download Training and Test Datasets

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/main.ipynb

Downloads the test and training datasets from S3 if they do not already exist locally. Ensure S3 paths are correctly configured.

```python
if not exists(TEST):
    s3.download(S3_TEST, TEST)
    s3.download(S3_TRAIN, TRAIN)
```

--------------------------------

### Download Model Weights

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/pack.ipynb

Downloads required model files from S3 if they do not already exist locally.

```python
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
    s3.download(S3_MODEL_SHAPE, MODEL_SHAPE)
    s3.download(S3_MODEL_ENCODER, MODEL_ENCODER)
    s3.download(S3_MODEL_NER, MODEL_NER)
```

--------------------------------

### Execute Jupyter Notebooks with Varying Parameters

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/grid.ipynb

This script iterates 100 times, generating random seeds and learning rates for a BERT model. It then executes a Jupyter notebook ('main.ipynb') with these parameters using nbconvert. Ensure 'main.ipynb' is in the same directory.

```python
# %run -n main.py
# seed(2)
# env = dict(environ)
# for index in range(100):
#     seed_ = str(randint(0, 100))
#     bert_lr = '%.7f' % (10 ** uniform(-6, -3))
#     lr = '%.5f' % (10 ** uniform(-4, -2))
#     lr_gamma = '%.2f' % uniform(0.1, 0.9)
#     board_name = f'{index:03d} {seed_} {bert_lr} {lr} {lr_gamma}'
#     env.update(
#         seed=seed_,
#         bert_lr=bert_lr,
#         lr=lr,
#         lr_gamma=lr_gamma,
#         board_name=board_name
#     )
#     run(
#         args=[
#             'jupyter', 'nbconvert',
#             '--ExecutePreprocessor.timeout=6000',
#             '--to=notebook',
#             '--execute', 'main.ipynb'
#         ],
#         env=env
#     )

```

--------------------------------

### Build and Run SlovNet NER Docker Service

Source: https://context7.com/natasha/slovnet/llms.txt

Builds a Docker image for the SlovNet NER service and runs it as a container, exposing port 8080. This is for production deployment.

```bash
# Build and run NER service
docker build -t slovnet-ner -f docker/slovnet-ner/exec/Dockerfile .
docker run -p 8080:8080 slovnet-ner

```

--------------------------------

### Download Essential Data Files

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/main.ipynb

Downloads the NE5 and FACTRU files from S3 if they do not already exist locally. This ensures that the required datasets are available for processing.

```python
if not exists(NE5):
    s3.download(S3_NE5, NE5)
    s3.download(S3_FACTRU, FACTRU)
```

--------------------------------

### Batch Dependency Parsing with Syntax.map

Source: https://context7.com/natasha/slovnet/llms.txt

Process multiple tokenized sentences in batches for dependency parsing. Requires Navec and a pre-trained Syntax model. Sentences are tokenized using razdel.

```python
from razdel import sentenize, tokenize
from navec import Navec
from slovnet import Syntax

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
syntax = Syntax.load('slovnet_syntax_news_v1.tar')
syntax.navec(navec)

text = 'Президент подписал указ. Документ вступает в силу немедленно.'

# Tokenize into sentences
chunk = []
for sent in sentenize(text):
    tokens = [_.text for _ in tokenize(sent.text)]
    chunk.append(tokens)

# Parse all sentences
for markup in syntax.map(chunk):
    ids = {token.id: token for token in markup.tokens}
    print('Dependencies:')
    for token in markup.tokens:
        head = ids.get(token.head_id)
        head_text = head.text if head else 'ROOT'
        print(f'  {token.text} --{token.rel}--> {head_text}')
    print()
```

--------------------------------

### Download Training Data

Source: https://github.com/natasha/slovnet/blob/master/scripts/03_bert_morph/main.ipynb

Downloads the NEWS and FICTION datasets from S3 if they do not already exist locally. This is a prerequisite for training.

```python
if not exists(NEWS):
    s3.download(S3_NEWS, NEWS)
    s3.download(S3_FICTION, FICTION)
```

--------------------------------

### Download and Load Navec Embeddings and Vocabularies

Source: https://github.com/natasha/slovnet/blob/master/scripts/06_morph/main.ipynb

Checks for the existence of NERUS and NAVEC data, downloading them from S3 or a URL if they are not present. It also downloads the TAGS_VOCAB file.

```python
if not exists(NERUS):
    s3.download(S3_NERUS, NERUS)
    
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
    s3.download(S3_TAGS_VOCAB, TAGS_VOCAB)
```

--------------------------------

### Load Packaged Model and Perform Inference

Source: https://github.com/natasha/slovnet/blob/master/scripts/06_morph/pack.ipynb

Loads a packaged morphological model and performs inference on a sample sentence. This demonstrates how to use the saved model for morphological analysis.

```python
morph = api.Morph.load(PACK)
morph.navec(navec)
words = 'Ежедневно очаги коронавирусной инфекции'.split()
markup = morph(words)
show_morph_markup(markup)
```

--------------------------------

### Process and Export Data

Source: https://github.com/natasha/slovnet/blob/master/scripts/02_bert_ner/data.ipynb

Iterates through settings to load, adapt, format, and save data as compressed JSONL files.

```python
for source, target, _, load in settings:
    records = load(source)
    records = log_progress(records, desc=load.__name__)
    records = (_.adapted for _ in records)
    items = (_.as_json for _ in records)
    lines = format_jl(items)
    dump_gz_lines(lines, target)
```

--------------------------------

### Download Data from S3 or wget

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/main.ipynb

Checks for the existence of NERUS and NAVEC data, downloading them from S3 or using wget if they are not present.

```python
if not exists(NERUS):
    s3.download(S3_NERUS, NERUS)
    s3.download(S3_RELS_VOCAB, RELS_VOCAB)
    
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
```

--------------------------------

### Upload Data to S3

Source: https://github.com/natasha/slovnet/blob/master/scripts/07_syntax/data.ipynb

This commented-out code snippet demonstrates uploading a file to an S3 bucket. Ensure the S3 client and necessary paths (NERUS, S3_NERUS) are configured.

```python
# s3.upload(NERUS, S3_NERUS)
```

--------------------------------

### Download Raw Datasets

Source: https://github.com/natasha/slovnet/blob/master/scripts/01_bert_news/data.ipynb

Downloads various news datasets from Yandex Cloud storage into the local raw directory.

```python
!wget https://storage.yandexcloud.net/natasha-corus/taiga/Fontanka.tar.gz -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/ods/gazeta_v1.csv.zip -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/ods/interfax_v1.csv.zip -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/lenta-ru-news.csv.gz -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/buriy/news-articles-2014.tar.bz2 -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/buriy/news-articles-2015-part1.tar.bz2 -P {RAW_DIR}
!wget https://storage.yandexcloud.net/natasha-corus/buriy/news-articles-2015-part2.tar.bz2 -P {RAW_DIR}
```

--------------------------------

### Morphology Tagging with SlovNet

Source: https://context7.com/natasha/slovnet/llms.txt

Loads Navec and Morph models, performs morphology tagging on a list of words, and prints token details. Requires Navec and Morph models to be downloaded.

```python
from navec import Navec
from slovnet import Morph

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
morph = Morph.load('slovnet_morph_news_v1.tar')
morph.navec(navec)

words = ['Красивый', 'город', 'Москва']
markup = morph(words)

# Access words and tags as lists
print(f'Words: {markup.words}')
print(f'Tags: {markup.tags}')

# Access individual token details
for token in markup.tokens:
    print(f'\nToken: {token.text}')
    print(f'  POS: {token.pos}')
    print(f'  Features: {token.feats}')
    print(f'  Full tag: {token.tag}')

# Convert to JSON
import json
print(f'\nJSON: {json.dumps(markup.as_json, ensure_ascii=False, indent=2)}')

```

--------------------------------

### Prepare Data for Training

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/main.ipynb

Loads and preprocesses the text data for training. It tokenizes the text, converts markups to BIO format, and encodes the data into batches suitable for the model. A subset of batches is selected for training and testing.

```python
lines = load_gz_lines(CUSTOM_TEXTS) if CUSTOM_TUNING else load_gz_lines(NERUS)
lines = log_progress(lines)

items = parse_jl(lines)

markups = (SpanMarkup.from_json(_) for _ in items)
markups = (_.to_bio(list(tokenize(_.text))) for _ in markups)

encode = TagTrainEncoder(
    words_vocab, shapes_vocab, tags_vocab,
    seq_len=256,
    batch_size=64,
    shuffle_size=1000,
)
batches = encode(markups)
# full lenta is ~10k batches
# epoch takes ~1h, on 2nd epoch loss expodes
# ~5k seems to saturate acc
batches = head(batches, 5000)
batches = [_.to(DEVICE) for _ in batches]

size = 10  # ~750 articles
batches = {
    TEST: batches[:size],
    TRAIN: batches[size:]
}
```

--------------------------------

### Load Syntax Dependency Parser with Syntax.load

Source: https://context7.com/natasha/slovnet/llms.txt

Load a syntax dependency parser model to identify head-dependent relations between tokens. Requires Navec and a pre-trained Syntax model. Input is a list of pre-tokenized words.

```python
from navec import Navec
from slovnet import Syntax

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
syntax = Syntax.load('slovnet_syntax_news_v1.tar')
syntax.navec(navec)

# Input is a list of tokens (pre-tokenized)
words = ['Опубликованы', 'новые', 'данные', 'по', 'заражению', 'коронавирусом', 'в', 'Москве']

markup = syntax(words)

# Access dependency relations
print('ID\tTOKEN\t\t\tHEAD_ID\tREL')
for token in markup.tokens:
    print(f'{token.id}\t{token.text:20}\t{token.head_id}\t{token.rel}')
```

--------------------------------

### Download and Unzip Dataset

Source: https://github.com/natasha/slovnet/blob/master/scripts/04_bert_syntax/data.ipynb

Downloads the GramEval2020 dataset from a specified URL and extracts its contents into the raw data directory. This is commented out by default.

```python
# !wget https://storage.yandexcloud.net/natasha-corus/GramEval2020-master.zip -P {RAW_DIR}
# !unzip {RAW_DIR}/GramEval2020-master.zip -d {RAW_DIR}
```

--------------------------------

### Remote Server Synchronization

Source: https://github.com/natasha/slovnet/blob/master/README.md

Commands to tunnel ports and synchronize files between local and remote GPU environments.

```bash
ssh -Nf gpu -L 8888:localhost:8888 -L 6006:localhost:6006

scp ~/.slovnet.json gpu:~
rsync --exclude data -rv . gpu:~/slovnet
rsync -u --exclude data -rv 'gpu:~/slovnet/*' .
```

--------------------------------

### Optimize Model for Execution

Source: https://github.com/natasha/slovnet/blob/master/scripts/05_ner/pack.ipynb

Prepares the model for production by stripping unnecessary components and separating arrays.

```python
model = model.to_exec()
model = model.strip_navec()
arrays, model = model.separate_arrays()
```

--------------------------------

### Download Data if Not Exists

Source: https://github.com/natasha/slovnet/blob/master/scripts/06_morph/pack.ipynb

Checks if the Navec data exists and downloads it along with vocabulary and model components from S3 if it's missing. Ensure S3 credentials and paths are correctly configured.

```python
if not exists(NAVEC):
    !wget {NAVEC_URL} -O {NAVEC}
    s3.download(S3_TAGS_VOCAB, TAGS_VOCAB)
    s3.download(S3_MODEL_SHAPE, MODEL_SHAPE)
    s3.download(S3_MODEL_ENCODER, MODEL_ENCODER)
    s3.download(S3_MODEL_MORPH, MODEL_MORPH)
```