### Seeded Protein Generation with ProtGPT2

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Generates protein sequences that are initiated with a specific amino acid or a short fragment. This allows for guiding the generation process towards sequences with desired starting characteristics or motifs.

```python
from transformers import pipeline

protgpt2 = pipeline('text-generation', model="nferruz/ProtGPT2")

# Generate sequences starting with methionine (M)
sequences = protgpt2(
    "M",
    max_length=100,
    do_sample=True,
    top_k=950,
    repetition_penalty=1.2,
    num_return_sequences=5,
    eos_token_id=0
)

# Generate sequences starting with a specific fragment
fragment = "MGLTT"
sequences_from_fragment = protgpt2(
    fragment,
    max_length=150,
    do_sample=True,
    top_k=950,
    repetition_penalty=1.2,
    num_return_sequences=3,
    eos_token_id=0
)

for seq in sequences_from_fragment:
    print(seq['generated_text'])
```

--------------------------------

### Finetune ProtGPT2 Model (Shell)

Source: https://github.com/teletchealab/protgpt2/blob/main/README.md

This command demonstrates how to finetune the ProtGPT2 model using the `run_clm.py` script from Hugging Face's transformers library. It requires specifying the model name, training and validation file paths, output directory, and learning rate. The finetuned model will be saved in the specified output directory.

```shell
python run_clm.py --model_name_or_path nferruz/ProtGPT2 --train_file training.txt --validation_file validation.txt --tokenizer_name nfergpt2 --do_train --do_eval --output_dir output --learning_rate 1e-06
```

--------------------------------

### ProtGPT2 Model Configuration Reference

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Demonstrates how to load and inspect the configuration of the ProtGPT2 model using the HuggingFace transformers library. This is useful for understanding the model's architecture and for creating custom implementations or modifications.

```python
from transformers import GPT2Config, GPT2LMHeadModel

# Load configuration
config = GPT2Config.from_pretrained("nferruz/ProtGPT2")

```

--------------------------------

### Initialize ProtGPT2 Pipeline for Sequence Generation

Source: https://github.com/teletchealab/protgpt2/blob/main/README.md

This snippet demonstrates how to initialize the ProtGPT2 model using the HuggingFace pipeline API. It sets up the text-generation pipeline with the pre-trained 'nferruz/ProtGPT2' model, enabling zero-shot protein sequence generation.

```python
from transformers import pipeline

# Initialize the text-generation pipeline with ProtGPT2
protgpt2 = pipeline('text-generation', model="nferruz/ProtGPT2")

# Example usage to generate a sequence starting with 'M'
sequences = protgpt2("M", max_length=100, num_return_sequences=1)
```

--------------------------------

### Load ProtGPT2 Model with HuggingFace Pipeline

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Demonstrates how to load the ProtGPT2 model using the HuggingFace transformers pipeline for text generation. The pipeline simplifies tokenization and model inference.

```python
from transformers import pipeline

# Load ProtGPT2 using the text-generation pipeline
protgpt2 = pipeline('text-generation', model="nferruz/ProtGPT2")

# Generate a single protein sequence
result = protgpt2("<|endoftext|>", max_length=100, do_sample=True, top_k=950, repetition_penalty=1.2)
print(result[0]['generated_text'])
```

--------------------------------

### Calculate and Display Model Parameter Count

Source: https://context7.com/teletchealab/protgpt2/llms.txt

This snippet shows how to define and format the total parameter count for the model, which is useful for logging and resource estimation during deployment.

```python
total_params = 738_000_000
print(f"Total parameters: {total_params:,}")
```

--------------------------------

### Fine-Tuning ProtGPT2 on Custom Protein Datasets

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Adapts the ProtGPT2 model for specific protein characteristics by fine-tuning on a custom dataset. This process involves preparing data in a specific format and using HuggingFace's run_clm.py script. The output is a fine-tuned model that can be used for directed generation.

```bash
# Prepare training data: Replace FASTA headers with <|endoftext|> token
# Split into training.txt (90%) and validation.txt (10%)

# Example training.txt format:
# <|endoftext|>
# MGLSDGEWQLVLNVWGKVEADIPGHGQEVLIRLFKGHPETLEKFDKFKHLKSEDEMKASE
# DLKKHGATVLTALGGILKKKGHHEAEIKPLAQSHATKHKIPVKYLEFISECIIQVLQSKH
# PGDFGADAQGAMNKALELFRKDMASNYKELGFQG
# <|endoftext|>
# MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSH-----GSAQ
# VKGHGKKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLP
# AEFTPAVHASLDKFLASVSTVLTSKYR

# Run fine-tuning with HuggingFace script
python run_clm.py \
    --model_name_or_path nferruz/ProtGPT2 \
    --train_file training.txt \
    --validation_file validation.txt \
    --tokenizer_name nferruz/ProtGPT2 \
    --do_train \
    --do_eval \
    --output_dir ./output \
    --learning_rate 1e-06 \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --save_steps 500

# After training, generate from the fine-tuned model
from transformers import pipeline
finetuned = pipeline('text-generation', model="./output")
sequences = finetuned("<|endoftext|", max_length=100, do_sample=True, top_k=950)
```

--------------------------------

### Loading ProtGPT2 Directly with Transformers

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Loads the ProtGPT2 model and tokenizer directly using the HuggingFace transformers library, allowing for fine-grained control over inference parameters and the creation of custom processing pipelines. This method is suitable for advanced users who need to integrate model generation into complex workflows.

```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained("nferruz/ProtGPT2")
tokenizer = GPT2Tokenizer.from_pretrained("nferruz/ProtGPT2")

# Move to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

# Encode input
input_text = "<|endoftext|>"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)

# Generate with custom parameters
with torch.no_grad():
    output = model.generate(
        input_ids,
        max_length=200,
        do_sample=True,
        top_k=950,
        top_p=0.95,
        temperature=1.0,
        repetition_penalty=1.2,
        num_return_sequences=5,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=0
    )

# Decode generated sequences
for i, sequence in enumerate(output):
    protein = tokenizer.decode(sequence, skip_special_tokens=True)
    print(f"Protein {i+1}: {protein}")
```

--------------------------------

### Inspect ProtGPT2 Architecture Configuration

Source: https://context7.com/teletchealab/protgpt2/llms.txt

This snippet demonstrates how to access and print key architectural parameters of the ProtGPT2 model configuration, such as the number of layers, embedding dimension, and attention heads. It provides insight into the model's capacity and structural constraints.

```python
print(f"Model type: {config.model_type}")
print(f"Layers: {config.n_layer}")
print(f"Embedding dimension: {config.n_embd}")
print(f"Attention heads: {config.n_head}")
print(f"Context length: {config.n_ctx}")
print(f"Vocabulary size: {config.vocab_size}")
print(f"Activation function: {config.activation_function}")
```

--------------------------------

### Generate Protein Sequences with ProtGPT2 (Python)

Source: https://github.com/teletchealab/protgpt2/blob/main/README.md

This snippet demonstrates how to use the ProtGPT2 model to generate protein sequences. It utilizes the `max_length`, `do_sample`, `top_k`, `repetition_penalty`, `num_return_sequences`, and `eos_token_id` parameters to control the generation process. The output is a list of generated sequences.

```python
sequences = protgpt2("<|endoftext|>", max_length=100, do_sample=True, top_k=950, repetition_penalty=1.2, num_return_sequences=10, eos_token_id=0)
for seq in sequences:
        print(seq)
```

--------------------------------

### Zero-Shot Protein Sequence Generation with ProtGPT2

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Generates novel protein sequences using ProtGPT2 in a zero-shot manner, meaning without any fine-tuning. This method leverages the model's pre-trained knowledge to produce sequences that adhere to natural protein distributions.

```python
from transformers import pipeline

protgpt2 = pipeline('text-generation', model="nferruz/ProtGPT2")

# Generate 10 novel protein sequences starting from the special token
# max_length is in tokens, where each token averages ~4 amino acids
sequences = protgpt2(
    "<|endoftext|>",
    max_length=100,
    do_sample=True,
    top_k=950,
    repetition_penalty=1.2,
    num_return_sequences=10,
    eos_token_id=0
)

# Print all generated sequences
for i, seq in enumerate(sequences):
    print(f"Sequence {i+1}: {seq['generated_text']}")
```

--------------------------------

### Prepare Sequence for Perplexity Calculation (Python)

Source: https://github.com/teletchealab/protgpt2/blob/main/README.md

This snippet shows how to format a protein sequence for perplexity calculation, which is often correlated with AlphaFold2's plddt. It involves defining the sequence and then wrapping it with the '<|endoftext|>' token at the beginning and end, and ensuring newlines are inserted every 60 amino acids to adhere to FASTA file format conventions.

```python
sequence='MGEAMGLTQPAVSRAVARLEERVGIRIFNRTARAITLTDEGRRFYEAVAPLLAGIEMHGY
VNVEGVAQLLELYARDILAEGRLVQLLPEWAD'

#Convert the sequence to a string like this
#(note we have to introduce new line characters every 60 amino acids,
#following the FASTA file format).

sequence = "<|endoftext|>MGEAMGLTQPAVSRAVARLEERVGIRIFNRTARAITLTDEGRRFYEAVAPLLAGIEMHGY
RVNVEGVAQLLELYARDILAEGRLVQLLPEWAD<|endoftext|>"
```

--------------------------------

### Batch Sequence Quality Assessment with ProtGPT2

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Evaluates multiple generated protein sequences using perplexity to rank them. It requires the transformers library and PyTorch. The input is a list of sequences, and the output is a ranked list of sequences with their perplexity scores.

```python
import torch
import math
from transformers import pipeline, GPT2LMHeadModel, GPT2Tokenizer

# Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = GPT2LMHeadModel.from_pretrained("nferruz/ProtGPT2").to(device)
tokenizer = GPT2Tokenizer.from_pretrained("nferruz/ProtGPT2")
protgpt2 = pipeline('text-generation', model="nferruz/ProtGPT2")

def calculate_perplexity(sequence, model, tokenizer):
    input_ids = torch.tensor(tokenizer.encode(sequence)).unsqueeze(0).to(device)
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    return math.exp(outputs[0])

def format_sequence(seq):
    """Format sequence with FASTA-style line breaks."""
    # Remove any existing formatting
    clean_seq = seq.replace('
', '').replace('<|endoftext|>', '')
    # Add line breaks every 60 characters
    formatted = '\n'.join([clean_seq[i:i+60] for i in range(0, len(clean_seq), 60)])
    return f"<|endoftext|>{formatted}<|endoftext|>"

# Generate many sequences
sequences = protgpt2(
    "<|endoftext|>",
```

```python
    max_length=100,
    do_sample=True,
    top_k=950,
    repetition_penalty=1.2,
    num_return_sequences=50,
    eos_token_id=0
)

# Score and rank all sequences
scored_sequences = []
for seq in sequences:
    formatted = format_sequence(seq['generated_text'])
    ppl = calculate_perplexity(formatted, model, tokenizer)
    scored_sequences.append({'sequence': seq['generated_text'], 'perplexity': ppl})

# Sort by perplexity (lower is better)
ranked = sorted(scored_sequences, key=lambda x: x['perplexity'])

# Select top candidates
print("Top 5 sequences by perplexity:")
for i, item in enumerate(ranked[:5]):
    print(f"{i+1}. Perplexity: {item['perplexity']:.2f}")
    print(f"   Sequence: {item['sequence'][:60]}...")
```

--------------------------------

### Calculate Protein Sequence Perplexity with ProtGPT2

Source: https://context7.com/teletchealab/protgpt2/llms.txt

Evaluates the quality of generated protein sequences by calculating their perplexity using the ProtGPT2 model and tokenizer. Lower perplexity scores indicate sequences that are more aligned with natural protein patterns and may correlate with higher structural prediction accuracy.

```python
import torch
import math
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load model and tokenizer
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = GPT2LMHeadModel.from_pretrained("nferruz/ProtGPT2").to(device)
tokenizer = GPT2Tokenizer.from_pretrained("nferruz/ProtGPT2")

def calculate_perplexity(sequence, model, tokenizer):
    """Calculate perplexity for a protein sequence."""
    input_ids = torch.tensor(tokenizer.encode(sequence)).unsqueeze(0)
    input_ids = input_ids.to(device)
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    return math.exp(loss)

# Format sequence with FASTA-style line breaks (60 amino acids per line)
# Wrap with special tokens
sequence = "<|endoftext|>MGEAMGLTQPAVSRAVARLEERVGIRIFNRTARAITLTDEGRRFYEAVAPLLAGIEMHGY\nRVNVEGVAQLLELYARDILAEGRLVQLLPEWAD<|endoftext|>"

ppl = calculate_perplexity(sequence, model, tokenizer)
print(f"Perplexity: {ppl}")
# Lower values indicate better sequence quality
```

--------------------------------

### Calculate Sequence Perplexity with ProtGPT2

Source: https://github.com/teletchealab/protgpt2/blob/main/README.md

This function computes the perplexity of a given sequence using a pre-trained model and tokenizer. It utilizes PyTorch to perform inference without gradient calculation, returning the exponentiated loss as the perplexity score.

```python
import torch
import math

def calculatePerplexity(sequence, model, tokenizer):
    input_ids = torch.tensor(tokenizer.encode(sequence)).unsqueeze(0)
    input_ids = input_ids.to(device)
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    return math.exp(loss)

# Usage example
ppl = calculatePerplexity(sequence, model, tokenizer)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.