### Shell: Managing lmcpp-server-cli

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Provides command-line examples for running the `lmcpp-server-cli` tool. It shows how to start the server with the Web UI enabled, using a default model, a model from a URL, or a local model file. These commands are executed within a Rust project context.

```sh
# With default model
cargo run --bin lmcpp-server-cli -- --webui
# Or with a specific model from URL:
cargo run --bin lmcpp-server-cli -- --webui -u https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf
# Or with a specific local model:
cargo run --bin lmcpp-server-cli -- --webui -l /path/to/local/model.gguf
```

--------------------------------

### GET /props

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Retrieves properties and configuration of the running llama.cpp server.

```APIDOC
## GET /props

### Description
Retrieves properties and configuration of the running llama.cpp server.

### Method
GET

### Endpoint
/props

### Parameters
None

### Request Example
```json
null
```

### Response
#### Success Response (200)
- **properties** (object) - An object containing various server properties and configurations.

#### Response Example
```json
{
  "properties": {
    "n_ctx": 2048,
    "n_gpu_layers": 0
  }
}
```
```

--------------------------------

### Initialize and Use LLM Prompts (Rust)

Source: https://github.com/shelbyjenkins/llm_client/blob/master/llm_prompt/README.md

Demonstrates initializing and using `LlmPrompt` for both API and local LLMs. It covers adding system, user, and assistant messages, setting generation prefixes for local models, accessing underlying prompt structures, and retrieving formatted prompts as strings or tokens. It also shows how to get total token counts and validate maximum token requests against model limits.

```rust
use llm_prompt::*;
use llm_models::api_model::CloudLlm;
use llm_models::local_model::LocalLlm;
use std::collections::HashMap;

// OpenAI Format
let model = CloudLlm::gpt_3_5_turbo();
let prompt = LlmPrompt::new_api_prompt(
    model.model_base.tokenizer.clone(),
    Some(model.tokens_per_message),
    model.tokens_per_name,
);

// Chat Template
let model = LocalLlm::default();
let prompt = LlmPrompt::new_local_prompt(
    model.model_base.tokenizer.clone(),
    &model.chat_template.chat_template,
    model.chat_template.bos_token.as_deref(),
    &model.chat_template.eos_token,
    model.chat_template.unk_token.as_deref(),
    model.chat_template.base_generation_prefix.as_deref(),
);

// Add system messages
prompt.add_system_message()?.set_content("You are a nice robot");

// User messages
prompt.add_user_message()?.set_content("Hello");

// LLM responses
prompt.add_assistant_message()?.set_content("Well, how do you do?");

// Builds with generation prefix. The llm will complete the response: 'Don't you think that is... cool?'
// Only Chat Template format supports this
prompt.set_generation_prefix("Don't you think that is...");

// Access (and build) the underlying prompt topography
let local_prompt: &LocalPrompt = prompt.local_prompt()?;
let api_prompt: &ApiPrompt = prompt.api_prompt()?;

// Get chat template formatted prompt
let local_prompt_as_string: String = prompt.local_prompt()?.get_built_prompt()?;
let local_prompt_as_tokens: Vec<u32> = prompt.local_prompt()?.get_built_prompt_as_tokens()?;

// Openai formatted prompt (Openai and Anthropic format)
let api_prompt_as_messages: Vec<HashMap<String, String>> = prompt.api_prompt()?.get_built_prompt()?;

// Get total tokens in prompt
let total_prompt_tokens: u64 = prompt.local_prompt()?.get_total_prompt_tokens();
let total_prompt_tokens: u64 = prompt.api_prompt()?.get_total_prompt_tokens();

// Validate requested max_tokens for a generation. If it exceeds the models limits, reduce max_tokens to a safe value
let actual_requested_response_tokens = check_and_get_max_tokens(
    model.context_length,
    Some(model.max_tokens_output), // If using a GGUF model use either model.context_length or the ctx_size of the server
    total_prompt_tokens,
    Some(10), // Safety tokens
    requested_max_tokens,
)?;

```

--------------------------------

### Shell: Launch Llama Server CLI with Web UI

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

These shell commands demonstrate how to launch the `lmcpp-server-cli` tool to start the llama.cpp server with the web UI enabled. You can specify a model either from a Hugging Face repository URL, a direct URL, or a local file path. This is useful for quick testing and interactive use of the llama server.

```sh
// With default model
cargo run --bin lmcpp-server-cli -- --webui
// Or with a specific model from URL:
cargo run --bin lmcpp-server-cli -- --webui -u https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf
// Or with a specific local model:
cargo run --bin lmcpp-server-cli -- --webui -l /path/to/local/model.gguf
```

--------------------------------

### Rust: Launch Llama Server and Get Completion

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

This Rust code snippet demonstrates how to use the lmcpp library to launch a llama.cpp server with a specified Hugging Face model and then perform a text completion task. It utilizes `LmcppServerLauncher` to configure and load the server, and the `completion` method on the `LmcppServer` to get a response. Ensure the 'lmcpp' crate is added as a dependency.

```rust
use lmcpp :: *;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")? 
                .build(),
        )
        .load()?;

    let res = server.completion(
        CompletionRequest::builder()
            .prompt("Tell me a joke about Rust.")
            .n_predict(64),
    )?;

    println!("Completion response: {:#?}", res.content);
    Ok(())
}
```

--------------------------------

### Launch Llama.cpp Server with Web UI Enabled (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Starts a llama.cpp server with HTTP transport and enables its built-in web interface. This allows for easier interaction and monitoring through a browser. The function also configures timeouts for model loading and downloading. Requires the `lmcpp` crate.

```rust
use lmcpp ::*;
use std::time::Duration;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .port(8080)
                .build(),
        )
        .webui(true) // Enable web UI
        .load_budget(Duration::from_secs(120)) // Allow 2 minutes for model loading
        .download_budget(Duration::from_secs(600)) // Allow 10 minutes for download
        .load()?;

    println!("Server running with Web UI at http://localhost:8080");

    // Keep server alive
    std::thread::sleep(Duration::from_secs(3600));

    Ok(())
}
```

--------------------------------

### Load Local GGUF Model (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Configures a loader for a locally stored GGUF model file using Rust. This example uses the `llm_models::gguf::loader::GgufLoader` and requires the `std::path::PathBuf` type to specify the file path. The loader is built with the local file path.

```rust
use llm_models::gguf::loader::GgufLoader;
use std::path::PathBuf;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load from local filesystem
    let loader = GgufLoader::builder()
        .from_local_path(PathBuf::from("/models/llama-7b-q4.gguf"))
        .build();

    println!("Local model path: {:?}", loader.from_local_path);

    Ok(())
}

```

--------------------------------

### Rust: LmcppServerLauncher Usage for Text Completion

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Demonstrates how to use `LmcppServerLauncher` to set up a `llama-server` with a specified model and perform text completion. It initializes the server with arguments, sends a completion request, and prints the response. This requires the `lmcpp` crate.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    let res = server.completion(
        CompletionRequest::builder()
            .prompt("Tell me a joke about Rust.")
            .n_predict(64),
    )?;

    println!("Completion response: {:#?}", res.content);
    Ok(())
}
```

--------------------------------

### OpenAI API Compatibility

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Provides compatibility with OpenAI API endpoints for seamless integration.

```APIDOC
## OpenAI API Compatibility

### Description
This section covers endpoints that are compatible with the OpenAI API. These allow for easier integration with existing applications designed for OpenAI's services.

### Method
Various (POST, GET, etc.)

### Endpoint
Various OpenAI-compatible routes (e.g., `/v1/chat/completions`)

### Parameters
Parameters will vary depending on the specific OpenAI endpoint being called. Refer to the OpenAI API documentation for details.

### Request Example
(Varies by endpoint. See OpenAI documentation)

### Response
#### Success Response (200)
Response structure will mimic the corresponding OpenAI API response.

#### Response Example
(Varies by endpoint. See OpenAI documentation)
```

--------------------------------

### POST /completion

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Generates text completions based on a given prompt and configuration.

```APIDOC
## POST /completion

### Description
Generates text completions based on a given prompt and configuration.

### Method
POST

### Endpoint
/completion

### Parameters
#### Request Body
- **prompt** (string) - Required - The text prompt to generate completions for.
- **n_predict** (integer) - Optional - The maximum number of tokens to predict.

### Request Example
```json
{
  "prompt": "Tell me a joke about Rust.",
  "n_predict": 64
}
```

### Response
#### Success Response (200)
- **content** (string) - The generated text completion.

#### Response Example
```json
{
  "content": "Why did the Rust programmer quit his job? Because he didn't get arrays!"
}
```
```

--------------------------------

### Completion Endpoint

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Generates a text completion based on the provided prompt and parameters.

```APIDOC
## POST /completion

### Description
Generates a text completion based on the provided prompt and parameters.

### Method
POST

### Endpoint
/completion

### Parameters
#### Request Body
- **prompt** (string) - Required - The text prompt to generate a completion for.
- **n_predict** (integer) - Optional - The maximum number of tokens to predict.

### Request Example
```json
{
  "prompt": "Tell me a joke about Rust.",
  "n_predict": 64
}
```

### Response
#### Success Response (200)
- **content** (string) - The generated text completion.

#### Response Example
```json
{
  "content": "Why did the Rust programmer break up with the C++ programmer? Because they had too many memory leaks!"
}
```
```

--------------------------------

### Configure and Run llama.cpp Toolchain in Rust

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

This Rust code snippet demonstrates how to configure a custom llama.cpp toolchain, specifying the release tag and backend (CPU, CUDA, or Metal). It then launches an inference server with specific model arguments and performs a text completion request. The toolchain is automatically downloaded and built if not already present, with multiple versions cached separately. Dependencies include the `lmcpp` crate.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    // Configure custom toolchain
    let toolchain = LmcppToolChain::builder()
        .release_tag("b4268") // Specific llama.cpp version
        .backend(Backend::Cpu) // CPU, CUDA, or Metal
        .build();

    let server = LmcppServerLauncher::builder()
        .toolchain(toolchain)
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // Toolchain automatically downloads and builds if needed
    // Multiple versions cached separately

    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Hello from a specific llama.cpp version!")
            .n_predict(50)
            .build(),
    )?;

    println!("{}", response.content);

    Ok(())
}
```

--------------------------------

### OpenAI API Endpoints

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Provides compatibility with OpenAI's API for various LLM tasks.

```APIDOC
## OpenAI API Endpoints

### Description
Exposes endpoints compatible with the OpenAI API specification, allowing existing OpenAI integrations to work with lmcpp.

### Method
POST (for most endpoints)

### Endpoint
Various, prefixed with `/v1/*` (e.g., `/v1/chat/completions`)

### Parameters
Request and response bodies will conform to the OpenAI API schema. Typically uses `serde_json::Value` for flexibility.

### Request Example (Illustrative for chat completions)
```json
{
  "model": "gpt-3.5-turbo",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"}
  ]
}
```

### Response
#### Success Response (200)
Response bodies will conform to the OpenAI API schema.

#### Response Example (Illustrative for chat completions)
```json
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "lmcpp-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The Los Angeles Dodgers won the World Series in 2020."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 15,
    "total_tokens": 25
  }
}
```
```

--------------------------------

### Create API-Style Prompts with llm_prompt

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Builds prompts formatted for OpenAI-style API models, including system, user, and assistant messages. Requires the llm_prompt crate and a custom tokenizer implementation conforming to the PromptTokenizer trait.

```rust
use llm_prompt::*
use std::sync::Arc;

// Assume you have a tokenizer implementation
struct MyTokenizer;
impl PromptTokenizer for MyTokenizer {
    fn tokenize(&self, input: &str) -> Vec<u64> {
        // Implementation details
        vec![]
    }
    fn count_tokens(&self, str: &str) -> u64 {
        str.split_whitespace().count() as u64
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut prompt = LlmPrompt::default();

    // Initialize for API-style prompts
    prompt.new_api_prompt(
        Arc::new(MyTokenizer),
        Some(3), // tokens per message
        Some(1), // tokens per name
    )?;

    // Build conversation
    prompt.add_system_message()?.set_content("You are a helpful assistant.");
    prompt.add_user_message()?.set_content("What is Rust?");
    prompt.add_assistant_message()?.set_content("Rust is a systems programming language.");
    prompt.add_user_message()?.set_content("Tell me more.");

    // Get formatted messages
    let messages = prompt.api_prompt()?.get_built_prompt()?;
    println!("API messages: {:?}", messages);

    // Get token count
    let token_count = prompt.get_total_prompt_tokens()?;
    println!("Total tokens: {}", token_count);

    Ok(())
}
```

--------------------------------

### Infill Endpoint

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Performs infilling of text based on a prompt and suffix.

```APIDOC
## POST /infill

### Description
Performs infilling of text based on a prompt and suffix.

### Method
POST

### Endpoint
/infill

### Parameters
#### Request Body
- **prompt** (string) - Required - The text prompt for infilling.
- **suffix** (string) - Required - The suffix text to complete the prompt.
- **n_predict** (integer) - Optional - The maximum number of tokens to predict.

### Request Example
```json
{
  "prompt": "The quick brown fox ",
  "suffix": " jumps over the lazy dog.",
  "n_predict": 64
}
```

### Response
#### Success Response (200)
- **content** (string) - The completed text.

#### Response Example
```json
{
  "content": "The quick brown fox jumps over the lazy dog."
}
```
```

--------------------------------

### POST /tokenize

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Converts a given text into a list of token IDs.

```APIDOC
## POST /tokenize

### Description
Converts a given text into a list of token IDs.

### Method
POST

### Endpoint
/tokenize

### Parameters
#### Request Body
- **text** (string) - Required - The text to tokenize.

### Request Example
```json
{
  "text": "Hello world"
}
```

### Response
#### Success Response (200)
- **tokens** (array of integers) - The token IDs corresponding to the input text.

#### Response Example
```json
{
  "tokens": [1, 2, 3, 4]
}
```
```

--------------------------------

### Create Local LLM Prompt with Chat Template (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Builds prompts using a model's native chat template format in Rust. It requires the `llm_prompt` crate and a custom tokenizer implementation. The function initializes a prompt, adds system and user messages, sets a generation prefix, and then retrieves the formatted prompt as text and tokens.

```rust
use llm_prompt::*
use std::sync::Arc;

struct MyTokenizer;
impl PromptTokenizer for MyTokenizer {
    fn tokenize(&self, input: &str) -> Vec<u64> {
        vec![]
    }
    fn count_tokens(&self, str: &str) -> u64 {
        str.split_whitespace().count() as u64
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut prompt = LlmPrompt::default();

    // Initialize with chat template
    let chat_template = "{% for message in messages %}{{ message.role }}: {{ message.content }}\n{% endfor %}";

    prompt.new_local_prompt(
        Arc::new(MyTokenizer),
        chat_template,
        Some("<s>"), // BOS token
        "</s>",      // EOS token
        None,        // UNK token
        Some("Assistant:"), // Generation prefix
    )?;

    // Build conversation
    prompt.add_system_message()?.set_content("You are a code expert.");
    prompt.add_user_message()?.set_content("Write a hello world in Rust.");

    // Set generation prefix to prime the response
    prompt.set_generation_prefix("Here's a simple example:");

    // Get formatted prompt
    let prompt_text = prompt.local_prompt()?.get_built_prompt()?;
    println!("Formatted prompt:\n{}", prompt_text);

    // Get as tokens
    let tokens = prompt.local_prompt()?.get_built_prompt_as_tokens()?;
    println!("Token count: {}", tokens.len());

    Ok(())
}

```

--------------------------------

### POST /detokenize

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Converts a list of token IDs back into human-readable text.

```APIDOC
## POST /detokenize

### Description
Converts a list of token IDs back into human-readable text.

### Method
POST

### Endpoint
/detokenize

### Parameters
#### Request Body
- **tokens** (array of integers) - Required - The token IDs to detokenize.

### Request Example
```json
{
  "tokens": [1, 2, 3, 4]
}
```

### Response
#### Success Response (200)
- **text** (string) - The detokenized text.

#### Response Example
```json
{
  "text": "Hello world"
}
```
```

--------------------------------

### Tokenize Endpoint

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Tokenizes a given text input.

```APIDOC
## POST /tokenize

### Description
Tokenizes a given text input.

### Method
POST

### Endpoint
/tokenize

### Parameters
#### Request Body
- **content** (string) - Required - The text content to tokenize.

### Request Example
```json
{
  "content": "Hello world"
}
```

### Response
#### Success Response (200)
- **tokens** (array of integers) - The token IDs for the input text.

#### Response Example
```json
{
  "tokens": [1, 2, 3, 4, 5]
}
```
```

--------------------------------

### Launch Llama.cpp Server with Default HuggingFace Model (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Launches a llama.cpp server with automatic model download and initialization using a specified HuggingFace repository. The server will be automatically cleaned up when the `server` object is dropped. This function requires the `lmcpp` crate.

```rust
use lmcpp ::*;

fn main() -> LmcppResult<()> {
    // Launch server with HuggingFace model
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")
                .expect("Failed to set HuggingFace repo")
                .build(),
        )
        .load()?;

    // Server is now running and ready for requests
    // Process will be automatically cleaned up when `server` is dropped

    Ok(())
}
```

--------------------------------

### Configure Advanced Sampling Parameters (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Fine-tunes text generation with advanced sampling parameters for quality and diversity control in Rust, using the `lmcpp` crate. This snippet demonstrates setting parameters like `temperature`, `top_k`, `top_p`, `repeat_penalty`, `frequency_penalty`, `presence_penalty`, and `stop` sequences for a completion request.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Explain machine learning:")
            .n_predict(150)
            // Sampling parameters
            .temperature(0.7)      // Randomness (0.0 = deterministic, 1.0+ = creative)
            .top_k(40)             // Consider top K tokens
            .top_p(0.95)           // Nucleus sampling threshold
            .repeat_penalty(1.1)   // Penalize repetition
            .frequency_penalty(0.5) // Reduce frequent token probability
            .presence_penalty(0.3)  // Encourage topic diversity
            // Stop sequences
            .stop(vec!["\n\n".to_string(), "###".to_string()])
            .build(),
    )?;

    println!("{}", response.content);

    Ok(())
}

```

--------------------------------

### Implement PromptTokenizer Trait (Rust)

Source: https://github.com/shelbyjenkins/llm_client/blob/master/llm_prompt/README.md

Shows how to implement the `PromptTokenizer` trait for a custom tokenizer, `LlmTokenizer`. This involves providing implementations for the `tokenize` and `count_tokens` methods, which are essential for `llm_prompt` to process text inputs.

```rust
use llm_prompt::PromptTokenizer;

struct LlmTokenizer;

impl PromptTokenizer for LlmTokenizer {
    fn tokenize(&self, input: &str) -> Vec<u32> {
        self.tokenize(input)
    }

    fn count_tokens(&self, str: &str) -> u32 {
        self.count_tokens(str)
    }
}

```

--------------------------------

### Load Local GGUF Model with Llama.cpp Server (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Launches a llama.cpp server configured to load a locally stored GGUF model file. This allows inference using models not available on HuggingFace. Parameters like context window size (`ctx_size`) and number of GPU layers (`n_gpu_layers`) can be specified. Requires the `lmcpp` crate.

```rust
use lmcpp ::*;
use std::path::PathBuf;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .model(PathBuf::from("/path/to/model.gguf"))
                .ctx_size(2048) // Context window size
                .n_gpu_layers(35) // GPU acceleration layers
                .build(),
        )
        .load()?;

    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Explain quantum computing")
            .n_predict(200)
            .build(),
    )?;

    println!("{}", response.content);

    Ok(())
}
```

--------------------------------

### Constrained Text Generation with BNF Grammar using lmcpp

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Generates text that conforms to a Backus-Naur Form (BNF) grammar specification. This allows for structured output generation. It requires the lmcpp crate and a model loaded via ServerLauncher.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // Grammar for generating JSON-like structures
    let grammar = r#"
        root ::= object
        object ::= "{" pair ("," pair)* "}"
        pair ::= string ":" value
        string ::= \" [a-zA-Z0-9_]+ \"
        value ::= string | number
        number ::= [0-9]+
    "#;

    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Generate a person object with name and age:")
            .grammar(grammar.to_string())
            .n_predict(50)
            .build(),
    )?;

    println!("Grammar-constrained output: {}", response.content);

    Ok(())
}
```

--------------------------------

### Load GGUF Model from HuggingFace (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Downloads and loads a GGUF model from a HuggingFace repository using Rust. This snippet utilizes the `llm_models::gguf::loader::GgufLoader` and requires a valid HuggingFace URL for the GGUF file. The loader is configured with the URL and then built.

```rust
use llm_models::gguf::loader::GgufLoader;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load model from HuggingFace URL
    let loader = GgufLoader::builder()
        .from_hf_quant_file_url(
            "https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf"
        )
        .build();

    println!("Model loader configured for: {:?}", loader.from_hf_file);

    Ok(())
}

```

--------------------------------

### POST /infill

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Performs infilling tasks, filling in the middle of a given text.

```APIDOC
## POST /infill

### Description
Performs infilling tasks, filling in the middle of a given text.

### Method
POST

### Endpoint
/infill

### Parameters
#### Request Body
- **prompt** (string) - Required - The text prompt with a missing section to fill.
- **n_predict** (integer) - Optional - The maximum number of tokens to predict for the infill.

### Request Example
```json
{
  "prompt": "The quick brown fox <FILL_ME> over the lazy dog.",
  "n_predict": 32
}
```

### Response
#### Success Response (200)
- **content** (string) - The generated text that fills the missing section.

#### Response Example
```json
{
  "content": "jumps"
}
```
```

--------------------------------

### Generate Text Completion with Llama.cpp Server (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Sends a text completion request to a running llama.cpp server to generate text based on a provided prompt. This function allows configuration of generation parameters like `n_predict`, `temperature`, `top_k`, and `top_p`. It requires the `lmcpp` crate and a running server instance.

```rust
use lmcpp ::*;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // Generate completion with builder pattern
    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Tell me a joke about Rust.")
            .n_predict(64) // Max 64 tokens
            .temperature(0.7)
            .top_k(40)
            .top_p(0.95)
            .build(),
    )?;

    println!("Generated text: {}", response.content);
    println!("Tokens generated: {}", response.tokens_predicted);

    Ok(())
}
```

--------------------------------

### POST /embeddings

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Generates embeddings (vector representations) for a given text.

```APIDOC
## POST /embeddings

### Description
Generates embeddings (vector representations) for a given text.

### Method
POST

### Endpoint
/embeddings

### Parameters
#### Request Body
- **text** (string) - Required - The text to generate embeddings for.

### Request Example
```json
{
  "text": "This is a test sentence."
}
```

### Response
#### Success Response (200)
- **embeddings** (array of numbers) - The generated embeddings for the input text.

#### Response Example
```json
{
  "embeddings": [0.123, -0.456, 0.789]
}
```
```

--------------------------------

### Stream Completion Responses (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Generates text with streaming output to process tokens as they arrive, using the `lmcpp` crate in Rust. This involves launching an `LmcppServer`, configuring completion requests with `stream(true)`, and then printing the final content. Note that actual streaming would involve processing partial responses.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // Enable streaming
    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Write a short story about a robot:")
            .n_predict(200)
            .stream(true) // Enable streaming mode
            .temperature(0.8)
            .build(),
    )?;

    // Note: In actual streaming, you would receive partial responses
    // This example shows the final result
    println!("Generated story:\n{}", response.content);

    Ok(())
}

```

--------------------------------

### Detokenize Endpoint

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Detokenizes a given list of token IDs.

```APIDOC
## POST /detokenize

### Description
Detokenizes a given list of token IDs.

### Method
POST

### Endpoint
/detokenize

### Parameters
#### Request Body
- **tokens** (array of integers) - Required - The token IDs to detokenize.

### Request Example
```json
{
  "tokens": [1, 2, 3, 4, 5]
}
```

### Response
#### Success Response (200)
- **content** (string) - The detokenized text.

#### Response Example
```json
{
  "content": "Hello world"
}
```
```

--------------------------------

### Server Status (Internal)

Source: https://github.com/shelbyjenkins/llm_client/blob/master/README.md

Provides internal health status of the llama.cpp server. This is an internal helper and not part of the standard API.

```APIDOC
## Internal Endpoint: status()

### Description
Provides internal health status of the llama.cpp server. This is an internal helper and not part of the standard API.

### Method
(Internal Helper)

### Endpoint
(Internal Helper)

### Parameters
None

### Request Example
(Not Applicable)

### Response
#### Success Response
- **status** (string) - The health status of the server (e.g., "ok", "error").

#### Response Example
```json
{
  "status": "ok"
}
```
```

--------------------------------

### Internal /status

Source: https://github.com/shelbyjenkins/llm_client/blob/master/lmcpp/README.md

Internal endpoint to check the health and status of the server.

```APIDOC
## Internal /status

### Description
Internal helper endpoint to check the health and status of the server.

### Method
GET (Internal)

### Endpoint
/status

### Parameters
None

### Request Example
```json
null
```

### Response
#### Success Response (200)
- **status** (string) - The current status of the server (e.g., "running").

#### Response Example
```json
{
  "status": "running"
}
```
```

--------------------------------

### JSON Schema Constrained Generation with lmcpp

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Generates text output that validates against a provided JSON schema. This ensures the generated content adheres to a specific structure. It requires the lmcpp crate and the serde_json crate for schema definition.

```rust
use lmcpp::*
use serde_json::json;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // JSON schema for structured output
    let schema = json!({
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer", "minimum": 0},
            "email": {"type": "string"}
        },
        "required": ["name", "age"]
    });

    let response = server.completion(
        CompletionRequest::builder()
            .prompt("Generate a user profile:")
            .json_schema(schema)
            .n_predict(100)
            .build(),
    )?;

    println!("JSON output: {}", response.content);

    Ok(())
}
```

--------------------------------

### Tokenize and Detokenize Text with lmcpp

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Converts text to token IDs and vice versa using the model's tokenizer. Requires the lmcpp crate and a specified Hugging Face repository for the model.

```rust
use lmcpp::*

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .build(),
        )
        .load()?;

    // Tokenize text to IDs
    let tokenize_response = server.tokenize(
        TokenizeRequest::builder()
            .content("Hello, world!")
            .build(),
    )?;

    println!("Token IDs: {:?}", tokenize_response.tokens);

    // Detokenize IDs back to text
    let detokenize_response = server.detokenize(
        DetokenizeRequest::builder()
            .tokens(tokenize_response.tokens)
            .build(),
    )?;

    println!("Reconstructed text: {}", detokenize_response.content);

    Ok(())
}
```

--------------------------------

### Generate Embeddings with Llama.cpp Server (Rust)

Source: https://context7.com/shelbyjenkins/llm_client/llms.txt

Enables embedding mode on the llama.cpp server and computes vector embeddings for a given text input. This is useful for semantic search and other NLP tasks that rely on text representations. Requires the `lmcpp` crate.

```rust
use lmcpp ::*;

fn main() -> LmcppResult<()> {
    let server = LmcppServerLauncher::builder()
        .server_args(
            ServerArgs::builder()
                .hf_repo("bartowski/google_gemma-3-1b-it-qat-GGUF")?
                .embedding(true) // Enable embedding mode
                .build(),
        )
        .load()?;

    let response = server.embeddings(
        EmbeddingsRequest::builder()
            .content("The quick brown fox jumps over the lazy dog")
            .build(),
    )?;

    println!("Embedding dimensions: {}", response.embedding.len());
    println!("First 5 values: {:?}", &response.embedding[..5]);

    Ok(())
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.