### Serve with vLLM

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Deploy the model as an OpenAI-compatible API server using vLLM. This setup is suitable for high-throughput inference and supports tool calling. Ensure vLLM version 0.11.1 or later is installed.

```bash
# Start vLLM server (requires vLLM 0.11.1+)
vllm serve arcee-ai/Trinity-Large-Preview \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

# Test the endpoint
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "arcee-ai/Trinity-Large-Preview",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.8,
    "max_tokens": 100
  }'

# Expected response:
# {
#   "choices": [{
#     "message": {
#       "role": "assistant",
#       "content": "The capital of France is Paris."
#     }
#   }]
# }
```

--------------------------------

### Integrate Trinity-Large-Preview with OpenRouter API

Source: https://github.com/arcee-ai/trinity-large-preview/blob/main/README.md

This example shows how to make a chat completion request to the Trinity-Large-Preview model via the OpenRouter API. Replace $OPENROUTER_API_KEY with your actual API key.

```bash
curl -X POST "https://openrouter.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "arcee-ai/trinity-large-preview",
    "messages": [
      {
        "role": "user",
        "content": "What are some fun things to do in New York?"
      }
    ]
  }'
```

--------------------------------

### Serve Model with VLLM

Source: https://github.com/arcee-ai/trinity-large-preview/blob/main/README.md

Deploy the model using the VLLM serving engine, supported in version 0.11.1 and later.

```bash
vllm serve arcee-ai/Trinity-Large-Preview \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

--------------------------------

### Run Trinity-Large-Preview with llama.cpp

Source: https://github.com/arcee-ai/trinity-large-preview/blob/main/README.md

Use this command to run the model locally with llama.cpp. Ensure you have version b7061 or later.

```bash
llama-server -hf arcee-ai/Trinity-Large-Preview-GGUF:q4_k_m
```

--------------------------------

### Run with llama.cpp

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Run a quantized GGUF version of the model locally using llama.cpp. This method is efficient for CPU and GPU inference on consumer hardware. Requires llama.cpp version b7061 or later.

```bash
# Start llama.cpp server with 4-bit quantized model (requires llama.cpp b7061+)
llama-server -hf arcee-ai/Trinity-Large-Preview-GGUF:q4_k_m

# Query the server
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a haiku about programming."}
    ]
  }'
```

--------------------------------

### Load Trinity-Large-Preview with Transformers

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Load the model and tokenizer using Hugging Face Transformers. Ensure `torch_dtype=torch.bfloat16` and `device_map="auto"` for efficient memory usage. `trust_remote_code=True` is required for the custom MoE architecture.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Large-Preview"

# Load tokenizer and model with automatic device placement
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True  # Required for custom AfmoeForCausalLM architecture
)

print(f"Model loaded with {model.config.num_experts} experts")
print(f"Active experts per token: {model.config.num_experts_per_tok}")
# Output: Model loaded with 256 experts
# Output: Active experts per token: 4
```

--------------------------------

### Tool Calling with Function Definitions

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Define external tools and apply them to the chat template to enable function calling capabilities.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import json

model_id = "arcee-ai/Trinity-Large-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

messages = [
    {"role": "user", "content": "What's the weather like in Paris?"}
]

# Apply chat template with tools
input_ids = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(input_ids, max_new_tokens=256)
response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)
```

--------------------------------

### Model Configuration Reference

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Access the model's architecture parameters using the AutoConfig class.

```python
from transformers import AutoConfig

config = AutoConfig.from_pretrained(
    "arcee-ai/Trinity-Large-Preview",
    trust_remote_code=True
)

# Key architecture parameters
print(f"Total hidden layers: {config.num_hidden_layers}")      # 60
print(f"Dense layers: {config.num_dense_layers}")               # 6
print(f"Total experts: {config.num_experts}")                   # 256
print(f"Experts per token: {config.num_experts_per_tok}")       # 4
print(f"Shared experts: {config.num_shared_experts}")           # 1
print(f"Hidden size: {config.hidden_size}")                     # 3072
print(f"Attention heads: {config.num_attention_heads}")         # 48
print(f"Key-value heads: {config.num_key_value_heads}")         # 8
print(f"Max position embeddings: {config.max_position_embeddings}")  # 262144
print(f"Sliding window size: {config.sliding_window}")          # 4096
print(f"Vocabulary size: {config.vocab_size}")                  # 200192
```

--------------------------------

### Load and Run Model with Transformers

Source: https://github.com/arcee-ai/trinity-large-preview/blob/main/README.md

Use the Hugging Face Transformers library to load the model and generate text responses. Requires setting trust_remote_code=True.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Large-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "Who are you?"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.8
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

--------------------------------

### Generate Text with Chat Template

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Generate conversational responses using the model's chat template. Apply the template to format messages and then use `model.generate` with sampling parameters for text generation. Decode the output using the tokenizer.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Large-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Create conversation with chat template
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain what a Mixture-of-Experts model is in simple terms."}
]

# Apply chat template and tokenize
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate response with sampling parameters
outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.8
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

--------------------------------

### Chat completion via OpenRouter API

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Use this cURL command to send chat requests to the Trinity Large Preview model hosted on OpenRouter.

```bash
curl -X POST "https://openrouter.ai/api/v1/chat/completions" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "arcee-ai/trinity-large-preview",
    "messages": [
      {
        "role": "user",
        "content": "What are some fun things to do in New York?"
      }
    ],
    "temperature": 0.8,
    "max_tokens": 500
  }'
```

--------------------------------

### Multi-turn Conversation

Source: https://context7.com/arcee-ai/trinity-large-preview/llms.txt

Maintain context by passing a list of message objects representing the conversation history to the model.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Large-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Multi-turn conversation history
messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "How do I read a JSON file in Python?"},
    {"role": "assistant", "content": "You can use the json module: `import json; data = json.load(open('file.json'))`"},
    {"role": "user", "content": "How do I handle errors if the file doesn't exist?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.8,
    top_p=0.8
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.