### Flexible LLM Completion Handle Operations in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

This Rust example shows how to work with completion handles provided by llama_cpp-rs for flexible token processing. It demonstrates fetching individual tokens, converting them to human-readable pieces, printing them incrementally, and optionally breaking early. It also shows how to convert the entire completion handle into a single string.

```rust
use llama_cpp::{
    LlamaModel,
    LlamaParams,
    SessionParams,
};
use llama_cpp::standard_sampler::StandardSampler;
use std::io::{self, Write};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session.advance_context("Count to 10:")
    .expect("Failed to advance context");

let mut handle = session.start_completing_with(StandardSampler::default(), 50)
    .expect("Failed to start completion");

// Get individual tokens
while let Some(token) = handle.next_token() {
    let piece = model.token_to_piece(token);
    print!("{}", piece);
    io::stdout().flush().unwrap();

    // Can break early based on conditions
    if piece.contains("10") {
        break;
    }
}
println!();

// Or convert entire handle to string
let mut session2 = session.clone();
let handle2 = session2.start_completing_with(StandardSampler::default(), 50)
    .expect("Failed to start");

let full_text = handle2.into_string();
println!("Full completion: {}", full_text);
```

--------------------------------

### Basic Text Generation in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Provides a simple example of text completion using the llama_cpp-rs library. It loads a model, creates a session, advances the context with a prompt, and then generates a sequence of tokens as text.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use std::io::{self, Write};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Add initial context
session.advance_context("The capital of France is")
    .expect("Failed to advance context");

// Generate completions as strings
let max_tokens = 100;
let mut completions = session.start_completing()
    .expect("Failed to start completion")
    .into_strings();

for (i, completion) in completions.enumerate() {
    if i >= max_tokens {
        break;
    }
    print!("{}", completion);
    let _ = io::stdout().flush();
}
println!();
```

--------------------------------

### Tokenization and Detokenization in Rust with llama_cpp

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Provides examples of how to convert text into token IDs and vice versa using the llama_cpp library. It demonstrates tokenizing a string, detokenizing individual tokens back to bytes (and then to string), and decoding a sequence of tokens into a complete string. It also shows how to retrieve special tokens like BOS, EOS, and Newline.

```rust
use llama_cpp::{LlamaModel, LlamaParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

// Tokenize text
let text = "Hello, world! How are you today?";
let tokens = model.tokenize_bytes(text, true, true)
    .expect("Failed to tokenize");

println!("Text: {}", text);
println!("Tokens: {:?}", tokens);
println!("Token count: {}", tokens.len());

// Detokenize individual tokens
for token in &tokens {
    let bytes = model.detokenize(*token);
    let text = String::from_utf8_lossy(bytes);
    println!("Token {:?} -> '{}'", token, text);
}

// Decode all tokens back to string
let decoded = model.decode_tokens(&tokens);
println!("Decoded: {}", decoded);

// Get special tokens
println!("BOS token: {:?}", model.bos());
println!("EOS token: {:?}", model.eos());
println!("Newline token: {:?}", model.nl());
```

--------------------------------

### Greedy Sampling for Deterministic Outputs in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Shows how to use a StandardSampler configured for greedy sampling, which ensures deterministic output by always selecting the most likely next token. The example loads a model, sets a context, and generates a single, predictable completion.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use llama_cpp::standard_sampler::StandardSampler;

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session.advance_context("The answer to 2 + 2 is")
    .expect("Failed to advance context");

// Greedy sampler always picks the most likely token
let sampler = StandardSampler::new_greedy();

let completions = session.start_completing_with(sampler, 50)
    .expect("Failed to start completion")
    .into_string();

println!("Result: {}", completions);
```

--------------------------------

### Load and Run LLM with llama_cpp-rs

Source: https://github.com/edgenai/llama_cpp-rs/blob/main/README.md

Demonstrates loading a GGUF model, creating a session, feeding context, and generating text completions using the llama_cpp-rs Rust crate. It requires the 'llama_cpp' crate and standard Rust IO operations.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams, StandardSampler};
use std::io::{self, Write};
use std::path::Path;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a model from anything that implements `AsRef<Path>`:
    let model = LlamaModel::load_from_file("path_to_model.gguf", LlamaParams::default())?;

    // A `LlamaModel` holds the weights shared across many _sessions_;
    let mut ctx = model.create_session(SessionParams::default())?;

    // You can feed anything that implements `AsRef<[u8]>` into the model's context.
    ctx.advance_context("This is the story of a man named Stanley.")?;

    // LLMs are typically used to predict the next word in a sequence. Let's generate some tokens!
    let max_tokens = 1024;
    let mut decoded_tokens = 0;

    // `ctx.start_completing_with` creates a worker thread that generates tokens.
    let mut completions = ctx.start_completing_with(StandardSampler::default(), 1024).into_strings();

    for completion in completions {
        print!("{completion}");
        io::stdout().flush()?;
        
        decoded_tokens += 1;
        
        if decoded_tokens > max_tokens {
            break;
        }
    }
    Ok(())
}
```

--------------------------------

### Custom Sampling with StandardSampler in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Demonstrates how to configure a StandardSampler with multiple stages like repetition penalty, TopK, TopP, MinP, and Temperature for creative text generation. It loads a model, advances the context, and then generates completions using the custom sampler.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use llama_cpp::standard_sampler::{StandardSampler, SamplerStage};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session.advance_context("Write a creative story about")
    .expect("Failed to advance context");

// Create custom sampler with specific parameters
let sampler = StandardSampler::new_softmax(
    vec![
        SamplerStage::RepetitionPenalty {
            repetition_penalty: 1.1,
            frequency_penalty: 0.0,
            presence_penalty: 0.0,
            last_n: 64,
        },
        SamplerStage::TopK(40),
        SamplerStage::TopP(0.95),
        SamplerStage::MinP(0.05),
        SamplerStage::Temperature(0.9),  // Higher temperature for creativity
    ],
    1  // min_keep
);

let mut completions = session.start_completing_with(sampler, 200)
    .expect("Failed to start completion")
    .into_strings();

for completion in completions {
    print!("{}", completion);
}
```

--------------------------------

### Load Model and Create Session in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Demonstrates how to load a GGUF model file and create one or more inference sessions using default parameters. This is the foundational step for performing any inference.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};

// Load a model from a GGUF file with default parameters
let model = LlamaModel::load_from_file(
    "models/llama-7b.gguf",
    LlamaParams::default()
).expect("Failed to load model");

// Create a session for inference
let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Sessions are lightweight - create multiple from the same model
let mut session2 = model.create_session(SessionParams::default())
    .expect("Failed to create second session");
```

--------------------------------

### Dynamic Temperature Sampling with StandardSampler in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Illustrates the use of dynamic temperature sampling, where the temperature adapts based on the model's confidence (entropy). This allows for more adaptive randomness in text generation. The snippet sets up a sampler with DynamicTemperature and TopP stages.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use llama_cpp::standard_sampler::{StandardSampler, SamplerStage};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session.advance_context("Explain quantum computing")
    .expect("Failed to advance context");

// Dynamic temperature adjusts based on model confidence
let sampler = StandardSampler::new_softmax(
    vec![
        SamplerStage::DynamicTemperature {
            min_temp: 0.5,
            max_temp: 1.5,
            exponent_val: 1.0,
        },
        SamplerStage::TopP(0.9),
    ],
    1
);

let completions = session.start_completing_with(sampler, 300)
    .expect("Failed to start completion")
    .into_string();

println!("{}", completions);
```

--------------------------------

### Async/Await Model Operations with Rust and Tokio

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Demonstrates how to perform Llama model operations asynchronously using Rust's async/await syntax with the Tokio runtime. This enables non-blocking operations for loading models, advancing context, and generating completions.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use tokio;

#[tokio::main]
async fn main() {
    // Load model asynchronously
    let model = LlamaModel::load_from_file_async(
        "models/llama-7b.gguf",
        LlamaParams::default()
    ).await.expect("Failed to load model");

    let mut session = model.create_session(SessionParams::default())
        .expect("Failed to create session");

    // Advance context asynchronously
    session.advance_context_async("Tell me a story")
        .await
        .expect("Failed to advance context");

    // Generate completions asynchronously
    let mut completions = session.start_completing()
        .expect("Failed to start completion");

    while let Some(token) = completions.next_token_async().await {
        let text = model.token_to_piece(token);
        print!("{}", text);
    }

    // Or get the full result as a string
    let mut session2 = model.create_session(SessionParams::default())
        .expect("Failed to create session");

    session2.advance_context_async("What is Rust?")
        .await
        .expect("Failed to advance");

    let result = session2.start_completing()
        .expect("Failed to start")
        .into_string_async()
        .await;

    println!("\nResult: {}", result);
}
```

--------------------------------

### Configure Model Parameters with GPU Acceleration in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Illustrates how to configure `LlamaParams` to enable GPU acceleration, memory mapping, and model splitting. This allows offloading layers to the GPU for improved performance.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SplitMode};

// Configure model to use GPU acceleration
let mut params = LlamaParams::default();
params.n_gpu_layers = 32;  // Offload 32 layers to GPU
params.main_gpu = 0;  // Use first GPU as main device
params.split_mode = SplitMode::Layer;  // Split layers across GPUs
params.use_mmap = true;  // Use memory mapping
params.use_mlock = true;  // Lock model in RAM

let model = LlamaModel::load_from_file(
    "models/llama-7b.gguf",
    params
).expect("Failed to load model with GPU acceleration");
```

--------------------------------

### Configure Session Parameters in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Shows how to customize `SessionParams` to control context size, threading, batching, random seed, and cache types. These parameters fine-tune the inference process.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams, CacheType, RopeScaling};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session_params = SessionParams::default();
session_params.n_ctx = 4096;  // Context window size
session_params.n_threads = 8;  // Threads for generation
session_params.n_threads_batch = 8;  // Threads for batch processing
session_params.n_batch = 512;  // Batch size for prompt processing
session_params.seed = 42;  // RNG seed for reproducibility
session_params.type_k = CacheType::F16;  // Key cache type
session_params.type_v = CacheType::F16;  // Value cache type
session_params.rope_scaling_type = RopeScaling::Linear;

let mut session = model.create_session(session_params)
    .expect("Failed to create session");
```

--------------------------------

### Stream LLM Completions with Rust Iterators

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

This Rust code demonstrates how to process generated tokens from an LLM as they arrive using iterator patterns. It shows how to iterate over raw tokens, collect completions into strings, or retrieve raw bytes. This approach is useful for real-time output display and conditional processing.

```rust
use llama_cpp::{
    LlamaModel,
    LlamaParams,
    SessionParams,
};
use llama_cpp::standard_sampler::StandardSampler;

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session.advance_context("List the first 5 prime numbers:")
    .expect("Failed to advance context");

let completions = session.start_completing_with(StandardSampler::default(), 100)
    .expect("Failed to start completion");

// Iterate over tokens
for token in completions {
    println!("Token: {:?}", token);
}

// Or convert to strings and process
let mut session2 = session.clone();
let strings = session2.start_completing_with(StandardSampler::default(), 100)
    .expect("Failed to start")
    .into_strings()
    .collect::<Vec<_>>();

println!("Generated {} string chunks", strings.len());

// Or get raw bytes
let mut session3 = session.clone();
let bytes = session3.start_completing_with(StandardSampler::default(), 100)
    .expect("Failed to start")
    .into_bytes()
    .collect::<Vec<_>>();

println!("Generated {} byte chunks", bytes.len());
```

--------------------------------

### Clone and Deep Copy Sessions in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Create independent copies of a session using either shallow cloning (`clone`) or deep copying (`deep_copy`). Shallow cloning shares the underlying context, while deep copying creates a fully independent session with its own state, allowing for divergent modifications.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session1 = model.create_session(SessionParams::default())
    .expect("Failed to create session");

session1.advance_context("The story begins")
    .expect("Failed to advance context");

// Shallow clone - shares underlying context
let session2 = session1.clone();

// Deep copy - creates independent session with same state
let mut session3 = session1.deep_copy()
    .expect("Failed to deep copy");

// session3 can be modified independently
session3.advance_context(" with a twist")
    .expect("Failed to advance");

println!("Original: {}", session1.context_size());
println!("Deep copy: {}", session3.context_size());

```

--------------------------------

### Monitor LLM Session Memory Usage in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

This Rust code illustrates how to monitor and track the memory consumption of an LLM session using the llama_cpp-rs library. It demonstrates checking the initial memory size, advancing the context to increase memory usage, and then checking the memory size again to observe the change.

```rust
use llama_cpp::{
    LlamaModel,
    LlamaParams,
    SessionParams,
};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Check initial memory
let initial_size = session.memory_size();
println!("Initial session size: {} bytes ({} MB)",
    initial_size,
    initial_size / 1024 / 1024
);

// Add context
session.advance_context("A very long context that will increase memory usage...")
    .expect("Failed to advance context");

// Check memory after adding context
let current_size = session.memory_size();
println!("Current session size: {} bytes ({} MB)",
    current_size,
    current_size / 1024 / 1024
);

println!("Memory increase: {} bytes", current_size - initial_size);
```

--------------------------------

### Manage Context with Tokens in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Directly work with token sequences to tokenize input, advance the model's context, and retrieve the current context. This method provides fine-grained control over the context by operating at the token level, allowing for explicit addition of special tokens like BOS.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams, Token};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Tokenize and add to context
let prompt = "The quick brown fox";
let tokens = model.tokenize_bytes(prompt, false, true)
    .expect("Failed to tokenize");

session.advance_context_with_tokens(&tokens)
    .expect("Failed to advance context");

println!("Context size: {}", session.context_size());

// Get current context
let current_tokens = session.context();
println!("Current context tokens: {:?}", current_tokens);

// Manually add BOS token
let bos_token = model.bos();
session.advance_context_with_tokens(&[bos_token])
    .expect("Failed to add BOS token");

```

--------------------------------

### Estimate Memory Usage for Llama Sessions in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Estimates the memory requirements for Llama model sessions and embeddings generation before creating them. This helps in resource planning. It depends on the 'llama_cpp' crate and GGuf model files.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

// Estimate session memory usage
let session_params = SessionParams::default();
let usage = model.estimate_session_size(&session_params);

println!("Estimated host memory: {} MB", usage.host_memory / 1024 / 1024);
println!("Estimated device memory: {} MB", usage.device_memory / 1024 / 1024);

// Estimate embeddings memory
let texts = ["sample text 1", "sample text 2"];
let tokens: Vec<_> = texts.iter()
    .map(|t| model.tokenize_bytes(t, true, false).unwrap())
    .collect();

let embed_params = EmbeddingsParams::default();
let embed_usage = model.estimate_embeddings_session_size(&tokens, &embed_params);

println!("Embeddings host memory: {} MB", embed_usage.host_memory / 1024 / 1024);
println!("Embeddings device memory: {} MB", embed_usage.device_memory / 1024 / 1024);
```

--------------------------------

### Set and Reset Context Efficiently in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Change the model's context efficiently by setting it to a new string or a sequence of tokens. The `set_context` method reuses shared prefixes to avoid re-processing them, while `set_context_to_tokens` allows direct manipulation using pre-tokenized input.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Initial context
session.advance_context("User: Hello\nAssistant: Hi there!")
    .expect("Failed to set initial context");

// Change to new context, reusing shared prefix
let new_context = "User: Hello\nAssistant: Hi there!\nUser: How are you?";
session.set_context(new_context)
    .expect("Failed to set new context");

// Only processes the new tokens, not the shared prefix
println!("New context size: {}", session.context_size());

// Set context from tokens directly
let tokens = model.tokenize_bytes("Different conversation", false, true)
    .expect("Failed to tokenize");
session.set_context_to_tokens(&tokens)
    .expect("Failed to set context from tokens");

```

--------------------------------

### Query Model Metadata and Properties in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Retrieves and displays metadata and properties of a loaded Llama model, such as vocabulary size, embedding length, context length, and special tokens. Requires the 'llama_cpp' crate and a GGuf model file.

```rust
use llama_cpp::{LlamaModel, LlamaParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

// Get model properties
println!("Vocabulary size: {}", model.vocabulary_size());
println!("Embedding length: {}", model.embed_len());
println!("Training context length: {}", model.train_len());
println!("Number of layers: {}", model.layers());

// Get special tokens
println!("BOS token: {:?}", model.bos());
println!("EOS token: {:?}", model.eos());
println!("Newline token: {:?}", model.nl());

// Infill tokens (for code completion models)
println!("Infill prefix token: {:?}", model.infill_prefix());
println!("Infill middle token: {:?}", model.infill_middle());
println!("Infill suffix token: {:?}", model.infill_suffix());
println!("End of text token: {:?}", model.eot());
```

--------------------------------

### Generate Text Embeddings with Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Generates vector embeddings for given text inputs using a specified Llama model. This is useful for semantic search and clustering. It requires the 'llama_cpp' crate and a GGuf model file.

```rust
use llama_cpp::{LlamaModel, LlamaParams, EmbeddingsParams};

let model = LlamaModel::load_from_file("models/embeddings-model.gguf", LlamaParams::default())
    .expect("Failed to load model");

// Configure embeddings parameters
let embed_params = EmbeddingsParams {
    n_threads: 8,
    n_threads_batch: 8,
};

// Generate embeddings for multiple texts
let texts = [
    "The quick brown fox jumps over the lazy dog",
    "Machine learning is a subset of artificial intelligence",
    "Rust is a systems programming language",
];

let embeddings = model.embeddings(&texts, embed_params)
    .expect("Failed to generate embeddings");

for (i, embedding) in embeddings.iter().enumerate() {
    println!("Text {}: {} dimensions", i, embedding.len());
    println!("First 5 values: {:?}", &embedding[..5.min(embedding.len())]);
}

// Calculate cosine similarity between embeddings
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
    let dot: f32 = a.iter().zip(b).map(|(x, y)| x * y).sum();
    let mag_a: f32 = a.iter().map(|x| x * x).sum::<f32>().sqrt();
    let mag_b: f32 = b.iter().map(|x| x * x).sum::<f32>().sqrt();
    dot / (mag_a * mag_b)
}

let sim = cosine_similarity(&embeddings[0], &embeddings[1]);
println!("Similarity: {:.4}", sim);
```

--------------------------------

### Truncate and Modify Context in Rust

Source: https://context7.com/edgenai/llama_cpp-rs/llms.txt

Modify the model's context by truncating it to a specific number of tokens or removing tokens within a given range. This is useful for managing conversation history length and editing specific parts of the context without re-tokenizing the entire input.

```rust
use llama_cpp::{LlamaModel, LlamaParams, SessionParams};

let model = LlamaModel::load_from_file("models/llama-7b.gguf", LlamaParams::default())
    .expect("Failed to load model");

let mut session = model.create_session(SessionParams::default())
    .expect("Failed to create session");

// Add some context
session.advance_context("First sentence. Second sentence. Third sentence.")
    .expect("Failed to advance context");

println!("Initial context size: {}", session.context_size());

// Truncate to keep only first N tokens
session.truncate_context(10)
    .expect("Failed to truncate");

println!("After truncation: {}", session.context_size());

// Remove tokens in a specific range
session.remove_tokens_in_range(5..8)
    .expect("Failed to remove range");

println!("After removal: {}", session.context_size());

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.