OxiLLaMa

https://docs.rs/oxillama/
Admin
OxiLLaMa is a pure Rust LLM inference engine serving as a sovereign alternative to llama.cpp,...
Tokens:4,840
Snippets:68
Trust Score:9.1
Update:2 months ago
Show doc for...
Tokens:
Raw
### GET /v1/models

Source: https://docs.rs/oxillama-server/%5E0.1.3

List loaded models.

```APIDOC
## GET /v1/models

### Description
List loaded models.

### Method
GET

### Endpoint
/v1/models
```

--------------------------------

### GET /v1/batches

Source: https://docs.rs/oxillama-server/%5E0.1.3

List batch jobs.

```APIDOC
## GET /v1/batches

### Description
List batch jobs.

### Method
GET

### Endpoint
/v1/batches
```

--------------------------------

### GET /admin/stats

Source: https://docs.rs/oxillama-server/%5E0.1.3

Server stats (admin).

```APIDOC
## GET /admin/stats

### Description
Server stats (admin).

### Method
GET

### Endpoint
/admin/stats
```

--------------------------------

### GET /v1/threads/:thread_id/runs/:run_id

Source: https://docs.rs/oxillama-server/%5E0.1.3

Get run status.

```APIDOC
## GET /v1/threads/:thread_id/runs/:run_id

### Description
Get run status.

### Method
GET

### Endpoint
/v1/threads/:thread_id/runs/:run_id

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
- **run_id** (string) - Required - Run identifier
```

--------------------------------

### GET /health

Source: https://docs.rs/oxillama-server/%5E0.1.3

Health check.

```APIDOC
## GET /health

### Description
Health check.

### Method
GET

### Endpoint
/health
```

--------------------------------

### GET /admin/loras

Source: https://docs.rs/oxillama-server/%5E0.1.3

List registered LoRA adapters (admin).

```APIDOC
## GET /admin/loras

### Description
List registered LoRA adapters (admin).

### Method
GET

### Endpoint
/admin/loras
```

--------------------------------

### GET /v1/batches/:id

Source: https://docs.rs/oxillama-server/%5E0.1.3

Retrieve batch job.

```APIDOC
## GET /v1/batches/:id

### Description
Retrieve batch job.

### Method
GET

### Endpoint
/v1/batches/:id

### Parameters
#### Path Parameters
- **id** (string) - Required - Batch job identifier
```

--------------------------------

### GET /admin/models

Source: https://docs.rs/oxillama-server/%5E0.1.3

List model pool (admin).

```APIDOC
## GET /admin/models

### Description
List model pool (admin).

### Method
GET

### Endpoint
/admin/models
```

--------------------------------

### GET /admin/health

Source: https://docs.rs/oxillama-server/%5E0.1.3

Extended health (admin).

```APIDOC
## GET /admin/health

### Description
Extended health (admin).

### Method
GET

### Endpoint
/admin/health
```

--------------------------------

### GET /v1/batches/:id/output

Source: https://docs.rs/oxillama-server/%5E0.1.3

Stream batch output JSONL.

```APIDOC
## GET /v1/batches/:id/output

### Description
Stream batch output JSONL.

### Method
GET

### Endpoint
/v1/batches/:id/output

### Parameters
#### Path Parameters
- **id** (string) - Required - Batch job identifier
```

--------------------------------

### GET /v1/threads/:thread_id

Source: https://docs.rs/oxillama-server/%5E0.1.3

Retrieve thread.

```APIDOC
## GET /v1/threads/:thread_id

### Description
Retrieve thread.

### Method
GET

### Endpoint
/v1/threads/:thread_id

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
```

--------------------------------

### GET /v1/threads/:thread_id/messages

Source: https://docs.rs/oxillama-server/%5E0.1.3

List thread messages.

```APIDOC
## GET /v1/threads/:thread_id/messages

### Description
List thread messages.

### Method
GET

### Endpoint
/v1/threads/:thread_id/messages

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
```

--------------------------------

### Initialize and Run Inference with OxiLLaMa

Source: https://docs.rs/oxillama/latest/oxillama/index.html

Demonstrates configuring the inference engine, loading a GGUF model, and generating text output.

```rust
use oxillama::runtime::{InferenceEngine, EngineConfig, SamplerConfig};

let config = EngineConfig {
    model_path: "model.gguf".to_string(),
    ..Default::default()
};
let mut engine = InferenceEngine::new(config);
engine.load_model().expect("failed to load model");
engine.generate("Hello", 128, |tok| print!("{tok}")).expect("generation failed");
```

--------------------------------

### InferenceEngine::generate

Source: https://docs.rs/oxillama/latest/oxillama

Initializes the inference engine with a configuration and performs text generation based on a prompt.

```APIDOC
## InferenceEngine::generate

### Description
Loads a GGUF model and generates text tokens based on the provided prompt and configuration.

### Method
Rust Method Call

### Parameters
- **config** (EngineConfig) - Required - Configuration object containing the model path.
- **prompt** (str) - Required - The input text for the model.
- **max_tokens** (usize) - Required - Maximum number of tokens to generate.
- **callback** (Fn) - Required - Closure to handle generated tokens.

### Example
```rust
use oxillama::runtime::{InferenceEngine, EngineConfig};

let config = EngineConfig {
    model_path: "model.gguf".to_string(),
    ..Default::default()
};
let mut engine = InferenceEngine::new(config);
engine.load_model().expect("failed to load model");
engine.generate("Hello", 128, |tok| print!("{tok}")).expect("generation failed");
```
```

--------------------------------

### Initialize GpuDispatcher and check for GPU availability

Source: https://docs.rs/oxillama-gpu/%5E0.1.3

Use GpuDispatcher to check if hardware acceleration is available. If the 'gpu' feature is disabled, the dispatcher will default to CPU fallback.

```rust
use oxillama_gpu::{GpuDispatcher, GpuContext};

let dispatcher = GpuDispatcher::new();
if dispatcher.has_gpu() {
    println!("GPU available — will use hardware acceleration");
} else {
    println!("No GPU — CPU fallback active");
}
```

--------------------------------

### POST /v1/threads

Source: https://docs.rs/oxillama-server/%5E0.1.3

Create Assistants API thread.

```APIDOC
## POST /v1/threads

### Description
Create Assistants API thread.

### Method
POST

### Endpoint
/v1/threads
```

--------------------------------

### POST /v1/batches

Source: https://docs.rs/oxillama-server/%5E0.1.3

Create batch job (disk-spooled).

```APIDOC
## POST /v1/batches

### Description
Create batch job (disk-spooled).

### Method
POST

### Endpoint
/v1/batches
```

--------------------------------

### Load a GGUF model

Source: https://docs.rs/oxillama-gguf/%5E0.1.3

Use GgufModel::load to initialize a model from a file path and access its metadata.

```rust
use oxillama_gguf::GgufModel;

let model = GgufModel::load("model.gguf").unwrap();
println!("Architecture: {}", model.architecture().unwrap());
println!("Tensors: {}", model.file.header.tensor_count);
```

--------------------------------

### POST /v1/completions

Source: https://docs.rs/oxillama-server/%5E0.1.3

Text completion endpoint.

```APIDOC
## POST /v1/completions

### Description
Text completion endpoint.

### Method
POST

### Endpoint
/v1/completions
```

--------------------------------

### POST /admin/models/load

Source: https://docs.rs/oxillama-server/%5E0.1.3

Background-load model (admin).

```APIDOC
## POST /admin/models/load

### Description
Background-load model (admin).

### Method
POST

### Endpoint
/admin/models/load
```

--------------------------------

### POST /v1/threads/:thread_id/runs

Source: https://docs.rs/oxillama-server/%5E0.1.3

Create and enqueue a run.

```APIDOC
## POST /v1/threads/:thread_id/runs

### Description
Create and enqueue a run.

### Method
POST

### Endpoint
/v1/threads/:thread_id/runs

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
```

--------------------------------

### POST /v1/chat/completions

Source: https://docs.rs/oxillama-server/%5E0.1.3

Chat completion endpoint.

```APIDOC
## POST /v1/chat/completions

### Description
Chat completion endpoint.

### Method
POST

### Endpoint
/v1/chat/completions
```

--------------------------------

### InferenceEngine::generate

Source: https://docs.rs/oxillama/latest/index.html

Generates text based on an input prompt using the loaded model.

```APIDOC
## InferenceEngine::generate

### Description
Generates text tokens based on a provided prompt string using the initialized inference engine.

### Signature
`engine.generate(prompt: &str, max_tokens: usize, callback: Fn(String)) -> Result<(), Error>`

### Parameters
- **prompt** (str) - The input text to generate from.
- **max_tokens** (usize) - The maximum number of tokens to generate.
- **callback** (Fn(String)) - A closure called for each generated token.

### Example
```rust
engine.generate("Hello", 128, |tok| print!("{tok}")).expect("generation failed");
```
```

--------------------------------

### InferenceEngine::generate

Source: https://docs.rs/oxillama

Generates text based on an input prompt using the loaded model.

```APIDOC
## InferenceEngine::generate

### Description
Generates text tokens based on a provided prompt string and a maximum token count, utilizing a callback function to process each generated token.

### Signature
`engine.generate(prompt: &str, max_tokens: usize, callback: Fn(String)) -> Result<(), Error>`

### Parameters
- **prompt** (str) - Required - The input text to generate from.
- **max_tokens** (usize) - Required - The maximum number of tokens to generate.
- **callback** (Fn(String)) - Required - A closure that receives each generated token as a string.

### Example
```rust
engine.generate("Hello", 128, |tok| print!("{tok}")).expect("generation failed");
```
```

--------------------------------

### POST /admin/loras

Source: https://docs.rs/oxillama-server/%5E0.1.3

Register a LoRA adapter (admin).

```APIDOC
## POST /admin/loras

### Description
Register a LoRA adapter (admin).

### Method
POST

### Endpoint
/admin/loras
```

--------------------------------

### InferenceEngine::generate

Source: https://docs.rs/oxillama/latest/oxillama/index.html

Generates text completions using the loaded model within the inference engine.

```APIDOC
## InferenceEngine::generate

### Description
Generates text based on an input prompt using the configured inference engine.

### Signature
`pub fn generate(&mut self, prompt: &str, max_tokens: usize, callback: impl FnMut(String)) -> Result<(), Error>`

### Parameters
- **prompt** (str) - The input text prompt for the model.
- **max_tokens** (usize) - The maximum number of tokens to generate.
- **callback** (FnMut) - A closure called for each generated token.

### Example
```rust
engine.generate("Hello", 128, |tok| print!("{tok}")).expect("generation failed");
```
```

--------------------------------

### InferenceEngine::generate

Source: https://docs.rs/oxillama/latest/src/oxillama/lib.rs.html

Performs text generation using the loaded model with the specified configuration.

```APIDOC
## InferenceEngine::generate

### Description
Generates text based on a prompt using the initialized inference engine.

### Method
Rust Method

### Signature
`engine.generate(prompt: &str, max_tokens: usize, callback: Fn(String)) -> Result<(), Error>`

### Parameters
- **prompt** (str) - Required - The input text to generate from.
- **max_tokens** (usize) - Required - The maximum number of tokens to generate.
- **callback** (Fn) - Required - A closure called for each generated token.

### Request Example
```rust
engine.generate("Hello", 128, |tok| print!("{tok}"));
```
```

--------------------------------

### Crate Module Re-exports

Source: https://docs.rs/oxillama/latest/oxillama/index.html

The primary re-exports for the OxiLLaMa API surface.

```rust
pub use oxillama_gguf as gguf;
```

```rust
pub use oxillama_quant as quant;
```

```rust
pub use oxillama_arch as arch;
```

```rust
pub use oxillama_runtime as runtime;
```

```rust
pub use oxillama_server as server;
```

```rust
pub use oxillama_bench as bench;
```

```rust
pub use oxillama_gpu as gpu;
```

--------------------------------

### POST /admin/models/unload

Source: https://docs.rs/oxillama-server/%5E0.1.3

Unload model (admin).

```APIDOC
## POST /admin/models/unload

### Description
Unload model (admin).

### Method
POST

### Endpoint
/admin/models/unload
```

--------------------------------

### POST /v1/embeddings

Source: https://docs.rs/oxillama-server/%5E0.1.3

Text embeddings endpoint.

```APIDOC
## POST /v1/embeddings

### Description
Text embeddings endpoint.

### Method
POST

### Endpoint
/v1/embeddings
```

--------------------------------

### POST /v1/threads/:thread_id/messages

Source: https://docs.rs/oxillama-server/%5E0.1.3

Append message to thread.

```APIDOC
## POST /v1/threads/:thread_id/messages

### Description
Append message to thread.

### Method
POST

### Endpoint
/v1/threads/:thread_id/messages

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
```

--------------------------------

### POST /v1/threads/:thread_id/runs/:run_id/cancel

Source: https://docs.rs/oxillama-server/%5E0.1.3

Cancel a run.

```APIDOC
## POST /v1/threads/:thread_id/runs/:run_id/cancel

### Description
Cancel a run.

### Method
POST

### Endpoint
/v1/threads/:thread_id/runs/:run_id/cancel

### Parameters
#### Path Parameters
- **thread_id** (string) - Required - Thread identifier
- **run_id** (string) - Required - Run identifier
```

--------------------------------

### POST /v1/batches/:id/cancel

Source: https://docs.rs/oxillama-server/%5E0.1.3

Cancel batch job.

```APIDOC
## POST /v1/batches/:id/cancel

### Description
Cancel batch job.

### Method
POST

### Endpoint
/v1/batches/:id/cancel

### Parameters
#### Path Parameters
- **id** (string) - Required - Batch job identifier
```

--------------------------------

### DELETE /admin/loras/{name}

Source: https://docs.rs/oxillama-server/%5E0.1.3

Unregister a LoRA adapter (admin).

```APIDOC
## DELETE /admin/loras/{name}

### Description
Unregister a LoRA adapter (admin).

### Method
DELETE

### Endpoint
/admin/loras/{name}

### Parameters
#### Path Parameters
- **name** (string) - Required - LoRA adapter name
```
Tokens:
Raw