### Install GigaCheck and Dependencies

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Install the GigaCheck package and its dependencies, including flash-attn for optimized attention mechanisms.

```bash
pip install -U setuptools
pip install -e . && pip install flash-attn==2.7.3 --no-build-isolation
```

--------------------------------

### CLI inference script

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Example of how to run inference using the command-line interface script. This demonstrates passing text and model paths as arguments to the `inference.py` script.

```APIDOC
## CLI inference script

```bash
TEXT="Where are we going? I asked. The old cab rattled through the icy night."
MODEL="iitolstykh/GigaCheck-Classifier-Multi"   # or a local checkpoint path

CUDA_VISIBLE_DEVICES="0" python3 gigacheck/inference/inference.py \
    --text "${TEXT}" \
    --model_path "${MODEL}" \
    --device "cuda:0"
# INFO | [model=iitolstykh/GigaCheck-Classifier-Multi] {'pred_label': 'human', 'classification_head_probs': [...]}
```
```

--------------------------------

### Convert Span Formats (Absolute to Center-Width)

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Converts spans from absolute (start, end) format to normalized (center, width) format used internally by the DETR head. Requires PyTorch.

```python
import torch
from gigacheck.model.src.interval_detector.span_utils import span_xx_to_cxw, span_cxw_to_xx

spans_xx = torch.tensor([[0.0, 1.0], [0.2, 0.4]])
spans_cxw = span_xx_to_cxw(spans_xx)
print(spans_cxw)
# tensor([[0.5000, 1.0000],
#         [0.3000, 0.2000]])

recovered = span_cxw_to_xx(spans_cxw)
print(recovered)
# tensor([[0.0000, 1.0000],
#         [0.2000, 0.4000]])
```

--------------------------------

### Merge Overlapping AI Span Predictions

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Merges overlapping or touching AI span predictions from `predict()` output using `IntervalTree`. Returns a sorted list of `(start, end)` integer tuples.

```python
import numpy as np
from gigacheck.inference.src.mistral_detector import MistralDetector, merge_intervals

# Suppose predict() returned these raw intervals (char_start, char_end, confidence)
raw = np.array([[10.0, 50.0, 0.91], [45.0, 80.0, 0.88], [200.0, 250.0, 0.95]])
merged = merge_intervals(raw)
print(merged)
```

--------------------------------

### Compute Temporal IoU Between Spans

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Computes pairwise intersection-over-union (IoU) and union between two sets of (start, end) spans. Used for span matching and metric computation during DETR training. Requires PyTorch.

```python
import torch
from gigacheck.model.src.interval_detector.span_utils import temporal_iou

spans1 = torch.tensor([[0.0, 0.2], [0.5, 1.0]])
spans2 = torch.tensor([[0.0, 0.3], [0.0, 1.0]])
iou, union = temporal_iou(spans1, spans2)
print(iou)
# tensor([[0.6667, 0.2000],
#         [0.0000, 0.5000]])
```

--------------------------------

### merge_intervals

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Merges overlapping or touching AI span predictions from the raw `ai_intervals` array. This utility function uses an `IntervalTree` to produce a sorted list of merged `(start, end)` integer tuples.

```APIDOC
## `merge_intervals` — merge overlapping AI span predictions

Takes the raw `ai_intervals` array returned by `predict()` and merges overlapping or touching intervals using an `IntervalTree`. Returns a sorted list of `(start, end)` integer tuples.

```python
import numpy as np
from gigacheck.inference.src.mistral_detector import MistralDetector, merge_intervals

# Suppose predict() returned these raw intervals (char_start, char_end, confidence)
raw = np.array([[10.0, 50.0, 0.91], [45.0, 80.0, 0.88], [200.0, 250.0, 0.95]])
merged = merge_intervals(raw)
print(merged)   # [(10, 80), (200, 250)]
```
```

--------------------------------

### Train DETR Model with Accelerate

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Launches the training script for a DETR model using accelerate for distributed training. Configure model, data paths, sequence lengths, model dimensions, loss functions, optimizer, and learning rate scheduler.

```bash
accelerate launch --num_processes 8 gigacheck/train/scripts/train_detr_model.py \
    --pretrained_model_name "mistralai/Mistral-7B-v0.3" \
    --train_data_path "/data/detection/train.jsonl" \
    --eval_data_path "/data/detection/valid.jsonl" \
    --extractor_dtype "bfloat16" \
    --max_sequence_length 1024 \
    --min_sequence_length 100 \
    --random_sequence_length True \
    --num_queries 45 \
    --dec_layers 3 \
    --enc_layers 3 \
    --dn_detr True \
    --aux_loss True \
    --model_dim 256 \
    --use_focal_loss True \
    --label_loss_coef 2.0 \
    --query_initialization_method "default" \
    --special_ref_points True \
    --output_dir "train_logs/mistral_7b_dn_detr" \
    --num_train_epochs 150 \
    --warmup_steps 100 \
    --lr_scheduler_type "cosine_with_min_lr" \
    --lr_scheduler_kwargs '{"min_lr_rate": 0.5}' \
    --learning_rate 0.0002 \
    --weight_decay 0.0001 \
    --optim "adamw_torch" \
    --per_device_train_batch_size 64 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --save_strategy "epoch" \
    --eval_strategy "epoch" \
    --eval_accumulation_steps 1 \
    --metric_for_best_model "eval_mAP@0.5-0.95" \
    --save_total_limit 2 \
    --logging_strategy "steps" \
    --logging_steps 1 \
    --seed 8888 \
    --dataloader_num_workers 8 \
    --gradient_checkpointing False \
    --report_to tensorboard
```

--------------------------------

### Load LoRA Checkpoint Programmatically

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Loads a PEFT LoRA adapter onto a base MistralAI model without merging weights. Useful for inference and adapter hot-swapping.

```python
import torch
from gigacheck.model.mistral_ai_detector import MistralAIDetectorForSequenceClassification
from gigacheck.model.src.model_load_utils import load_lora_ckpt

base_model = MistralAIDetectorForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.3",
    num_labels=2,
    id2label={0: "ai", 1: "human"},
    torch_dtype=torch.bfloat16,
)
model_with_lora = load_lora_ckpt(
    lora_path="train_logs/mistral_7b_cls/checkpoint-3120",
    model=base_model,
)
model_with_lora.eval()
```

--------------------------------

### Run CLI Inference Script

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Executes the Gigacheck inference script from the command line. Specify text, model path, and device.

```bash
TEXT="Where are we going? I asked. The old cab rattled through the icy night."
MODEL="iitolstykh/GigaCheck-Classifier-Multi"   # or a local checkpoint path

CUDA_VISIBLE_DEVICES="0" python3 gigacheck/inference/inference.py \
    --text "${TEXT}" \
    --model_path "${MODEL}" \
    --device "cuda:0"
```

--------------------------------

### Train DETR Detection Model with Accelerate

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Trains the DN-DETR span detection head with a frozen Mistral backbone using Accelerate for multi-GPU training. Requires specific data format with 'ai_char_intervals'.

```bash
export TOKENIZERS_PARALLELISM="false"

accelerate launch --num_processes 8 gigacheck/train/scripts/train_detr_model.py \
    --pretrained_model_name "mistralai/Mistral-7B-v0.3" \
    --train_data_path "/data/detection/train.jsonl" \
    --eval_data_path  "/data/detection/valid.jsonl" \
    --extractor_dtype "bfloat16" \
    --max_sequence_length 1024 \
    --min_sequence_length 100 \
    --random_sequence_length True \
    --num_queries 45 \
    --dec_layers 3 \
    --enc_layers 3 \
    --dn_detr True \
    --aux_loss True \
    --model_dim 256 \
    --use_focal_loss True \
    --label_loss_coef 2.0 \
    --special_ref_points True \
    --output_dir "train_logs/mistral_7b_dn_detr" \
    --num_train_epochs 150 \
    --learning_rate 0.0002 \
    --weight_decay 0.0001 \
    --optim "adamw_torch" \
    --per_device_train_batch_size 64 \
    --metric_for_best_model "eval_mAP@0.5-0.95" \
    --save_strategy "epoch" \
    --eval_strategy "epoch" \
    --seed 8888 \
    --report_to tensorboard
```

--------------------------------

### Evaluate Classification Model Checkpoint

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Runs the HuggingFace Trainer evaluation loop on a pre-trained and merged classification model checkpoint. Logs per-class accuracy metrics to the console.

```bash
python3 gigacheck/train/scripts/eval_classification_model.py \
    --pretrained_model_path "train_logs/mistral_7b_cls/final_model" \
    --eval_data_path        "/data/classification/test.jsonl"
# INFO | eval/mean_cls_accuracy:  0.9312
# INFO | eval/ai_accuracy:        0.9501
# INFO | eval/human_accuracy:     0.9124
```

--------------------------------

### Initialize MistralDetector for Inference

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Initializes MistralDetector, a high-level inference wrapper. Loads tokenizer and model using `from_pretrained`. Configure `with_detr` based on model type.

```python
from transformers import AutoConfig
from gigacheck.inference.src.mistral_detector import MistralDetector

model_path = "iitolstykh/GigaCheck-Detector-Multi"   # or a local path
config = AutoConfig.from_pretrained(model_path)

detector = MistralDetector(
    max_seq_len=config.max_length,
    with_detr=config.with_detr,   # True for detection models
    id2label=config.id2label,
    device="cuda:0",
    conf_interval_thresh=0.8,     # minimum confidence to keep a predicted AI span
).from_pretrained(model_path)

text = (
    "Where are we going? I asked. The old cab rattled stiffly through the icy Montclair night. "
    "Shut up, said the cabby. Shut the fuck up."
)
output = detector.predict(text)

# Classification-only model output:
# {'pred_label': 'human', 'classification_head_probs': array([0.04, 0.96])}

# Detection model output:
# {
#   'pred_label': 'mixed',
#   'classification_head_probs': array([0.12, 0.03, 0.85]),
#   'ai_intervals': array([[42.1, 110.7, 0.93]])   # [char_start, char_end, confidence]
# }
print(output)
```

--------------------------------

### Save TextSample list to a .jsonl file

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Serialize a list of TextSample objects to a .jsonl file in the specified output directory.

```python
from gigacheck.train.src.data.data_format import TextSample, save_samples_jsonl

samples = [
    TextSample(label="human", model="human", text="Hello world.", data_type="misc"),
    TextSample(label="ai",    model="claude-3", text="Hello universe.", data_type="misc"),
]
save_samples_jsonl(samples, filename="my_dataset", out_dir="/tmp/data")
# >> Saved 2 texts to /tmp/data/my_dataset.jsonl
```

--------------------------------

### Perform Inference with GigaCheck

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Runs the inference script to detect LLM-generated content. Requires specifying the text to analyze and the path to the pre-trained model.

```bash
CUDA_VISIBLE_DEVICES="0" \
python3 gigacheck/inference/inference.py \
--text "${TEXT}" \
--model_path ${model_path}
```

--------------------------------

### Load a detector model

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Loads a pre-trained Mistral backbone with a DETR head for sequence classification and detection. The model can be configured with various options like freezing the backbone and specifying the number of labels.

```APIDOC
## Load a detector model
```python
from gigacheck.model.src.mistral_detector import MistralAIDetectorForSequenceClassification
import torch

detector = MistralAIDetectorForSequenceClassification.from_pretrained(
    pretrained_model_name_or_path="iitolab/GigaCheck-Detector-Multi",
    with_detr=True,
    freeze_backbone=True,
    num_labels=3,
    id2label={0: "ai", 1: "human", 2: "mixed"},
    torch_dtype=torch.float32,
)
detector.eval()
```
```

--------------------------------

### Save Processed Data to Files

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Saves the prepared training, validation, and testing datasets to their respective JSONL files in a specified directory. This step finalizes the preprocessing pipeline.

```python
path = "/data/TriBERT/OOD/8/"
save_to_jsonl(train_jsonl, path + "train.jsonl")
save_to_jsonl(valid_jsonl, path + "valid.jsonl")
save_to_jsonl(test_jsonl, path + "test.jsonl")
```

--------------------------------

### Load dataset from .jsonl file or directory

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Use the Corpus class to load TextSample objects from a single .jsonl file or multiple .jsonl files within a directory.

```python
from gigacheck.train.src.data.corpus import Corpus

# From a single file
corpus = Corpus("/data/classification/train.jsonl")
print(len(corpus))          # e.g. 45000

# From a directory of .jsonl files
corpus_dir = Corpus("/data/classification/")
print(corpus_dir.name)      # directory stem
for sample in corpus_dir.data[:3]:
    print(sample.label, sample.text[:40])
```

--------------------------------

### Load Mistral AIDetector Model

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Loads a pre-trained Mistral backbone with a DETR head for sequence classification. Set `with_detr=True` for detection models.

```python
from transformers import AutoConfig
from gigacheck.model.src.mistral_aidetector import MistralAIDetectorForSequenceClassification
import torch

detector = MistralAIDetectorForSequenceClassification.from_pretrained(
    pretrained_model_name_or_path="iitolstykh/GigaCheck-Detector-Multi",
    with_detr=True,
    freeze_backbone=True,
    num_labels=3,
    id2label={0: "ai", 1: "human", 2: "mixed"},
    torch_dtype=torch.float32,
)
detector.eval()
```

--------------------------------

### Configure DetrModelConfig

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Constructs a DetrModelConfig dataclass for DETR span localization head hyperparameters. Supports construction from a dictionary and round-tripping.

```python
from gigacheck.model.src.interval_detector.config import DetrModelConfig

cfg = DetrModelConfig(
    model_dim=256,
    num_queries=45,
    dec_layers=3,
    enc_layers=3,
    dn_detr=True,
    aux_loss=True,
    use_focal_loss=True,
    label_loss_coef=2.0,
    special_ref_points=True,
    extractor_dtype="bfloat16",
)
print(cfg.to_dict())

# Round-trip from dict
cfg2 = DetrModelConfig.from_dict(cfg.to_dict())
assert cfg2.num_queries == 45
```

--------------------------------

### Split Data into Train, Validation, and Test Sets

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Splits the processed JSONL data into training, validation, and testing sets based on the 'original_split' field. This is a standard procedure for preparing data for supervised learning.

```python
jsonl_out = convert_func(df)

train_jsonl = []
valid_jsonl = []
test_jsonl = []

for json_str in jsonl_out:
    if json_str['original_split'] == 'train':
        train_jsonl.append(json_str)
    elif json_str['original_split'] == 'valid':
        valid_jsonl.append(json_str)
    elif json_str['original_split'] == 'test':
        test_jsonl.append(json_str)
```

--------------------------------

### Evaluate DETR Detection Model

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Script to evaluate a DETR detection model. Specify the path to the pretrained model and the evaluation dataset.

```bash
python3 gigacheck/train/scripts/eval_detr_model.py \
    --pretrained_model_path "train_logs/mistral_7b_dn_detr/checkpoint-best" \
    --eval_data_path        "/data/detection/test.jsonl"
# INFO | eval_mAP@0.5-0.95: 0.487
```

--------------------------------

### Train Classification Model with DeepSpeed

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Command to initiate the training of a classification model using DeepSpeed. It specifies model paths, dataset locations, sequence lengths, LoRA configuration, and training hyperparameters.

```bash
deepspeed gigacheck/train/scripts/train_classification_model.py \
    --deepspeed ${ROOT_DIR}/gigacheck/deepspeed_configs/zero2.json \
    --pretrained_model_name "mistralai/Mistral-7B-v0.3" \
    --attn_implementation "flash_attention_2" \
    --train_data_path "/data/classification/train.jsonl" \
    --eval_data_path "/data/classification/valid.jsonl" \
    --max_sequence_length 1024 \
    --min_sequence_length 100 \
    --random_sequence_length True \
    --lora_enable True \
    --lora_r 8 \
    --bf16 True \
    --output_dir "train_logs/mistral_7b_cls" \
    --num_train_epochs 20 \
    --learning_rate 0.00003 \
    --lr_scheduler_type "cosine_with_min_lr" \
    --lr_scheduler_kwargs '{"min_lr_rate": 0.5}' \
    --warmup_steps 20 \
    --optim "adamw_torch" \
    --per_device_train_batch_size 64 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --eval_accumulation_steps 1 \
    --metric_for_best_model "eval/mean_cls_accuracy" \
    --save_strategy "steps" \
    --eval_strategy "steps" \
    --save_steps 81 \
    --eval_steps 81 \
    --save_total_limit 3 \
    --logging_strategy "steps" \
    --logging_steps 1 \
    --seed 8888 \
    --dataloader_num_workers 8 \
    --report_to tensorboard \
    --gradient_checkpointing False \
    --torch_compile False \
    --load_best_model_at_end False \
    --full_determinism True
```

--------------------------------

### Define TextSample for Classification and Detection

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Define classification and detection samples using the TextSample format. For detection, include 'ai_char_intervals'. Write samples to a .jsonl file.

```python
# Classification sample (label is "ai" or "human")
import json

cls_sample = {
    "label": "ai",
    "model": "gpt-4",
    "text": "The quick brown fox jumps over the lazy dog.",
    "data_type": "news"
}

# Detection sample (label is "mixed")
det_sample = {
    "label": "mixed",
    "model": "gpt-3.5-turbo",
    "text": "This opening is human-written. The rest was generated by an AI model to complete the paragraph.",
    "data_type": "news",
    "ai_char_intervals": [[32, 91]]   # character offsets of the AI-written portion
}

# Writing a .jsonl file
with open("train.jsonl", "w") as f:
    f.write(json.dumps(cls_sample) + "\n")
    f.write(json.dumps(det_sample) + "\n")
```

--------------------------------

### Run Classifier-Only Inference with MistralDetector

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Performs inference using a classifier-only MistralDetector model. Ensure `with_detr` is set to `False`.

```python
# Classifier-only example
model_path = "iitolstykh/GigaCheck-Classifier-Multi"
from transformers import AutoConfig
from gigacheck.inference.src.mistral_detector import MistralDetector

config = AutoConfig.from_pretrained(model_path)
clf = MistralDetector(
    max_seq_len=config.max_length,
    with_detr=False,
    id2label=config.id2label,
    device="cuda:0",
).from_pretrained(model_path)

result = clf.predict("This text was fully written by a human author.")
print(result["pred_label"])
print(result["classification_head_probs"])
```

--------------------------------

### Train Classification Model with DeepSpeed and LoRA

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Fine-tunes a Mistral-7B model using LoRA adapters for binary classification. Requires DeepSpeed and specific data paths. Adjust LoRA parameters and training configurations as needed.

```bash
export TOKENIZERS_PARALLELISM="false"

deepspeed gigacheck/train/scripts/train_classification_model.py \
    --deepspeed gigacheck/deepspeed_configs/zero2.json \
    --pretrained_model_name "mistralai/Mistral-7B-v0.3" \
    --attn_implementation "flash_attention_2" \
    --train_data_path "/data/classification/train.jsonl" \
    --eval_data_path  "/data/classification/valid.jsonl" \
    --max_sequence_length 1024 \
    --min_sequence_length 100 \
    --random_sequence_length True \
    --lora_enable True \
    --lora_r 8 \
    --lora_alpha 16 \
    --lora_dropout 0.1 \
    --bf16 True \
    --output_dir "train_logs/mistral_7b_cls" \
    --num_train_epochs 20 \
    --learning_rate 0.00003 \
    --lr_scheduler_type "cosine_with_min_lr" \
    --lr_scheduler_kwargs '{"min_lr_rate": 0.5}' \
    --warmup_steps 20 \
    --optim "adamw_torch" \
    --per_device_train_batch_size 64 \
    --per_device_eval_batch_size 1 \
    --metric_for_best_model "eval/mean_cls_accuracy" \
    --save_strategy "steps" \
    --eval_strategy "steps" \
    --save_steps 81 \
    --eval_steps 81 \
    --save_total_limit 3 \
    --seed 8888 \
    --report_to tensorboard
```

--------------------------------

### Load Data from Excel File

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Loads data from a specified Excel file into a pandas DataFrame and displays the first few rows.

```python
file_path = '/data/TriBERT-Out_of_Domain_Evaluation/data.xlsx'
df = pd.read_excel(file_path)
df.head()
```

--------------------------------

### Stream TextSample objects from a .jsonl file

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Use read_jsonl_dataset to lazily iterate over TextSample objects from a .jsonl file. Raises ValueError on missing keys.

```python
from gigacheck.train.src.data.data_format import read_jsonl_dataset

for sample in read_jsonl_dataset("train.jsonl"):
    print(sample.label, sample.model, sample.text[:60])
```

--------------------------------

### Import Libraries for Data Processing

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Imports necessary libraries for data manipulation and analysis, including pandas for dataframes, json for JSON handling, and ast for abstract syntax trees.

```python
import pandas as pd
import json
import ast
```

--------------------------------

### Load MistralAIDetectorForSequenceClassification model

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Load a pre-trained Mistral-7B model with an attached ClassificationHead and optional DETR interval detector. The backbone can be frozen.

```python
import torch
from gigacheck.model.mistral_ai_detector import MistralAIDetectorForSequenceClassification

# Load a fine-tuned classifier (no DETR)
model = MistralAIDetectorForSequenceClassification.from_pretrained(
    pretrained_model_name_or_path="iitolstykh/GigaCheck-Classifier-Multi",
    num_labels=2,
    id2label={0: "ai", 1: "human"},
    torch_dtype=torch.bfloat16,
)
model.eval()
```

--------------------------------

### Merge LoRA Weights for Classification Model

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Script to merge LoRA weights into the base model after classification training. Requires paths to the LoRA checkpoint and the model configuration.

```bash
python3 gigacheck/train/merge_lora_weights.py \
--lora_ckpt_path "train_logs/mistral_7b_cls/checkpoint-3120" \
--config_path "train_logs/mistral_7b_cls/config.json" \
--output_path "train_logs/mistral_7b_cls/final_model"
```

--------------------------------

### Merge LoRA Adapter Weights into Base Model

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Merges trained LoRA adapter weights into the base Mistral-7B model for a self-contained checkpoint. Use this after classification training.

```bash
python3 gigacheck/train/scripts/merge_lora_weights.py \
    --lora_ckpt_path "train_logs/mistral_7b_cls/checkpoint-3120" \
    --config_path    "train_logs/mistral_7b_cls/config.json" \
    --output_path    "train_logs/mistral_7b_cls/final_model"
```

```python
# Equivalent Python API
from gigacheck.train.scripts.merge_lora_weights import join_adapter

join_adapter(
    adapter_path="train_logs/mistral_7b_cls/checkpoint-3120",
    config_path="train_logs/mistral_7b_cls/config.json",
    output_path="train_logs/mistral_7b_cls/final_model",
)
# INFO | Saved to train_logs/mistral_7b_cls/final_model
```

--------------------------------

### Save Data to JSONL File

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Defines a utility function to save a list of dictionaries to a JSONL file. It handles file writing with UTF-8 encoding and includes basic error handling.

```python
def save_to_jsonl(data, filename):
    try:
        with open(filename, 'w', encoding='utf-8') as file:
            for item in data:
                file.write(json.dumps(item) + '\n')
        print(f"Data save in {filename}")
    except Exception as e:
        print(f"Mistake while saving in {filename}: {e}")
```

--------------------------------

### DetrModelConfig

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Configuration for the DETR head, including hyperparameters for span localization. This dataclass can be initialized from a dictionary and supports round-trip conversion.

```APIDOC
## `DetrModelConfig` — DETR head configuration

Dataclass holding all hyperparameters for the DN-DETR span localization head. Can be constructed from a plain dict (e.g., as stored in the HuggingFace config).

```python
from gigacheck.model.src.interval_detector.config import DetrModelConfig

cfg = DetrModelConfig(
    model_dim=256,
    num_queries=45,
    dec_layers=3,
    enc_layers=3,
    dn_detr=True,
    aux_loss=True,
    use_focal_loss=True,
    label_loss_coef=2.0,
    special_ref_points=True,
    extractor_dtype="bfloat16",
)
print(cfg.to_dict())
# {'extractor_dtype': 'bfloat16', 'model_dim': 256, 'num_queries': 45, ...}

# Round-trip from dict
cfg2 = DetrModelConfig.from_dict(cfg.to_dict())
assert cfg2.num_queries == 45
```
```

--------------------------------

### Compute Generalized Temporal IoU for Spans

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Computes Generalized IoU (GIoU) for span matching, used as the GIoU loss term during DETR training. Values range from -1 to 1, where 1 indicates a perfect match. Requires PyTorch.

```python
import torch
from gigacheck.model.src.interval_detector.span_utils import generalized_temporal_iou

spans1 = torch.tensor([[0.0, 0.2], [0.5, 1.0]])
spans2 = torch.tensor([[0.0, 0.3], [0.0, 1.0]])
giou = generalized_temporal_iou(spans1, spans2)
print(giou)
# tensor([[ 0.6667,  0.2000],
#         [-0.2000,  0.5000]])
```

--------------------------------

### MistralDetector

Source: https://context7.com/ai-forever/gigacheck/llms.txt

High-level inference wrapper for Mistral models. It handles tokenization, sequence truncation, and post-processing. It can load both the tokenizer and the model using `from_pretrained()` and provides a `predict(text)` method.

```APIDOC
## `MistralDetector` — high-level inference wrapper

Wraps `MistralAIDetectorForSequenceClassification` with tokenization, sequence truncation, and post-processing. Calls `from_pretrained()` to load both the tokenizer and the model, then exposes a single `predict(text)` method.

```python
from transformers import AutoConfig
from gigacheck.inference.src.mistral_detector import MistralDetector

model_path = "iitolstykh/GigaCheck-Detector-Multi"   # or a local path
config = AutoConfig.from_pretrained(model_path)

detector = MistralDetector(
    max_seq_len=config.max_length,
    with_detr=config.with_detr,   # True for detection models
    id2label=config.id2label,
    device="cuda:0",
    conf_interval_thresh=0.8,     # minimum confidence to keep a predicted AI span
).from_pretrained(model_path)

text = (
    "Where are we going? I asked. The old cab rattled stiffly through the icy Montclair night. "
    "Shut up, said the cabby. Shut the fuck up."
)
output = detector.predict(text)

# Classification-only model output:
# {'pred_label': 'human', 'classification_head_probs': array([0.04, 0.96])}

# Detection model output:
# {
#   'pred_label': 'mixed',
#   'classification_head_probs': array([0.12, 0.03, 0.85]),
#   'ai_intervals': array([[42.1, 110.7, 0.93]])   # [char_start, char_end, confidence]
# }
print(output)
```
```

--------------------------------

### Dataset Format for Classification Training

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Defines the expected JSON Lines format for datasets used in training classification models. Each line must contain label, model, text, and data_type.

```json
{
    "label": "human", 
    "model": "human", 
    "text": "...", 
    "data_type": "news"
}
```

--------------------------------

### Dataset Format for DETR Training

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

Specifies the JSON Lines format for datasets used in training DETR models. Includes 'ai_char_intervals' for character-level AI detection.

```json
{
    "label": "mixed", 
    "model": "gpt-3.5-turbo", 
    "text": "...", 
    "data_type": "news",
    "ai_char_intervals": [[492, 1003]]
}
```

--------------------------------

### GigaCheck Citation

Source: https://github.com/ai-forever/gigacheck/blob/main/README.md

BibTeX entries for citing the GigaCheck project and related research papers.

```bibtex
@article{Layer2025LLMTrace,
  Title = {{LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text}},
  Author = {Irina Tolstykh and Aleksandra Tsybina and Sergey Yakubson and Maksim Kuprashevich},
  Year = {2025},
  Eprint = {arXiv:2509.21269}
}
@article{tolstykh2024gigacheck,
  title={{GigaCheck: Detecting LLM-generated Content via Object-Centric Span Localization}},
  author={Irina Tolstykh and Aleksandra Tsybina and Sergey Yakubson and Aleksandr Gordeev and Vladimir Dokholyan and Maksim Kuprashevich},
  journal={arXiv preprint arXiv:2410.23728},
  year={2024}
}
```

--------------------------------

### MistralDetector.predict

Source: https://context7.com/ai-forever/gigacheck/llms.txt

Runs inference on a single text. This method tokenizes the input text, passes it through the model, and returns a dictionary containing the predicted label, classification probabilities, and AI intervals (for DETR models).

```APIDOC
## `MistralDetector.predict` — run inference on a single text

Tokenizes `text`, feeds it through the model, and returns a dict with `pred_label`, `classification_head_probs`, and (for DETR models) `ai_intervals` as an `np.ndarray` of shape `(N, 3)` where columns are `[char_start, char_end, ai_confidence]`.

```python
# Classifier-only example
model_path = "iitolstykh/GigaCheck-Classifier-Multi"
from transformers import AutoConfig
from gigacheck.inference.src.mistral_detector import MistralDetector

config = AutoConfig.from_pretrained(model_path)
clf = MistralDetector(
    max_seq_len=config.max_length,
    with_detr=False,
    id2label=config.id2label,
    device="cuda:0",
).from_pretrained(model_path)

result = clf.predict("This text was fully written by a human author.")
print(result["pred_label"])
# "human"
print(result["classification_head_probs"])
# e.g. array([0.02, 0.98])
```
```

--------------------------------

### Split Data Excluding a Specific Essay Set

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Splits the processed JSONL data into training, validation, and testing sets, while excluding a specified 'essayset' from the training and validation sets and including it only in the test set. This is useful for out-of-distribution testing.

```python
essayset = 8 # set from 1 to 8
jsonl_out = convert_func(df)

train_jsonl = []
valid_jsonl = []
test_jsonl = []


for json_str in jsonl_out:
    if json_str['original_split'] == 'train' and json_str['essayset'] != essayset:
        train_jsonl.append(json_str)
    elif json_str['original_split'] == 'valid' and json_str['essayset'] != essayset:
        valid_jsonl.append(json_str)
    elif json_str['original_split'] == 'test' and json_str['essayset'] == essayset:
        test_jsonl.append(json_str)
```

--------------------------------

### Calculate Character and Word Intervals

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Calculates character and word intervals for 'machine' labeled data and determines separator indices based on the specified split type. This function is crucial for segmenting and analyzing text data.

```python
def calculate_intervals(data, split_type):
    char_position = 0
    word_position = 0

    char_intervals = []
    word_intervals = []
    sep_indices = []

    for text, label in data:
        word_count = len(text.split())
        char_count = len(text)
        
        if label == 'machine':
            char_start = char_position
            char_end = char_position + char_count
            
            word_start = word_position
            word_end = word_position + word_count - 1

            if char_intervals and char_intervals[-1][1] + 1 == char_start:
                char_intervals[-1] = [char_intervals[-1][0], char_end]
                word_intervals[-1] = [word_intervals[-1][0], word_end]
            else:
                char_intervals.append([char_start, char_end])
                word_intervals.append([word_start, word_end])

        char_position += char_count + 1
        word_position += word_count
    
    if split_type == "H_M":
        sep_indices.append(char_intervals[0][0])
    elif split_type == "M_H":
        sep_indices.append(char_intervals[0][1])
    elif split_type == "H_M_H":
        sep_indices.append(char_intervals[0][0])
        sep_indices.append(char_intervals[0][1])
    elif split_type == "M_H_M":
        sep_indices.append(char_intervals[0][1])
        sep_indices.append(char_intervals[1][0])
    elif split_type == "H_M_H_M":
        sep_indices.append(char_intervals[0][0])
        sep_indices.append(char_intervals[0][1])
        sep_indices.append(char_intervals[1][0])
    elif split_type == "M_H_M_H":
        sep_indices.append(char_intervals[0][1])
        sep_indices.append(char_intervals[1][0])
        sep_indices.append(char_intervals[1][1])
            
    return char_intervals, word_intervals, sep_indices
```

--------------------------------

### Convert DataFrame to JSONL Format

Source: https://github.com/ai-forever/gigacheck/blob/main/data_scripts/detection/TriBERT_preprocessing.ipynb

Converts a pandas DataFrame into a list of JSON objects, creating entries for human and mixed text types. It calculates interval data for mixed texts. Use this function to transform the raw dataset into a format ready for further processing or saving.

```python
def convert_func(dataframe):
    
    out_jsonl = []
    for i in range(len(dataframe)):
        
        data_human = {
            "label": "human",
            "model": "human",
            "text":  dataframe.iloc[i]['human_part'],
            "source": "TriBERT",
            "source_dataset": "TriBERT",
            "essayset": int(dataframe.iloc[i]['essayset']),
            "data_type": "article",
            "original_split": dataframe.iloc[i]['train_ix'],
            "topic_id": "empty",
            "prompt": "empty",
            "ended": True,
        }
        
        data_ai = {
            "label": "ai",
            "model": "gpt",
            "text":  dataframe.iloc[i]['machine_part'],
            "source": "TriBERT",
            "source_dataset": "TriBERT",
            "essayset": int(dataframe.iloc[i]['essayset']),
            "data_type": "article",
            "original_split": dataframe.iloc[i]['train_ix'],
            "topic_id": "empty",
            "prompt": "empty",
            "prompt_type": "machine_specified",
            "ended": True,
        }
        
        text = eval(df.iloc[i]['sent_and_label'])
        split_type = df.iloc[i]['author_seq']
        ai_char_intervals, ai_words_intervals, sep_indices = calculate_intervals(text, split_type)
        
        data_mixed = {
            "label": "mixed",
            "model": "gpt",
            "text":  dataframe.iloc[i]['hybrid_text'],
            "source": "TriBERT",
            "source_dataset": "TriBERT",
            "essayset": int(dataframe.iloc[i]['essayset']),
            "data_type": "article",
            "original_split": dataframe.iloc[i]['train_ix'],
            "topic_id": "empty",
            "prompt": "empty",
            "prompt_type": "machine_specified",
            "ended": True,
            "ai_char_intervals": ai_char_intervals,
            "boundary_ix": ast.literal_eval(dataframe.iloc[i]['boundary_ix']),
            "sep_indices": sep_indices,
            "split": eval(dataframe.iloc[i]['sent_and_label']),
        }
        
        out_jsonl.append(data_human)
        # out_jsonl.append(data_ai)
        out_jsonl.append(data_mixed)
    
    return out_jsonl
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.