AbstractCore

https://github.com/lpalbou/abstractcore
Admin
Unified LLM interface for Python: one create_llm() API across local runtimes (MLX, llama.cpp,...
Tokens:818,419
Snippets:4,966
Trust Score:7.2
License:MIT
Update:2 days ago
Show doc for...
Tokens:
Raw
### Start vLLM server with LoRA support (production)

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Production setup with tensor parallelism and dynamic LoRA adapter loading. Enables up to 4 LoRA adapters.

```bash
# Multiple GPUs + LoRA support (Production setup)
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --enable-lora --max-loras 4 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 8192 \
    --max-num-seqs 128
```

--------------------------------

### Test Setup with vLLM

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Basic generation and guided JSON with vLLM-specific feature.

```python
from abstractcore import create_llm

# Basic generation
llm = create_llm("vllm", model="Qwen/Qwen3-Coder-30B-A3B-Instruct")
response = llm.generate("Write a Python function to sort a list")
print(response.content)

# With guided JSON (vLLM-specific feature)
response = llm.generate(
    "List 3 programming languages",
    guided_json={
        "type": "object",
        "properties": {
            "languages": {"type": "array", "items": {"type": "string"}}
        }
    }
)
print(response.content)
```

--------------------------------

### First call with OpenAI

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/getting-started.md

Shows a basic generation call using the OpenAI provider. Requires the light install and setting the OPENAI_API_KEY environment variable.

```python
from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")
resp = llm.generate("What is the capital of France?")
print(resp.content)
```

--------------------------------

### Install and run async demo

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/async-guide.md

Install the package and run the educational async demo with the ollama provider and qwen3:4b model.

```bash
pip install abstractcore

# Try the educational async demo
python examples/cli/async_cli_demo.py --provider ollama --model qwen3:4b
```

--------------------------------

### Download Ollama Models

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Pull a model and verify it is installed.

```bash
# Pull any model you want to use, then verify it's installed.
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama list
```

--------------------------------

### Start AbstractCore server

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/embeddings.md

Install the abstractcore package and start the server to use embeddings via REST API.

```bash
# Start the server
pip install abstractcore
abstractcore serve
```

--------------------------------

### Install and run local model with abstractcore serve

Source: https://github.com/lpalbou/abstractcore/blob/main/README.md

Install the package with Apple silicon extras and start the local server. The printed link opens the web console for model download and engine installation.

```bash
pip install "abstractcore[apple]"    # Apple silicon; "abstractcore[gpu]" on NVIDIA / AMD
abstractcore serve
# open the printed link: http://127.0.0.1:8000/console#claim=...
```

--------------------------------

### Start Ollama Service

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Start the Ollama server in the background.

```bash
# Start Ollama server (runs in background)
ollama serve
```

--------------------------------

### Install and run the abstractcore server

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/api.md

Installs the abstractcore package and starts the OpenAI-compatible /v1 gateway server. Run these commands in sequence to launch the server.

```bash
pip install abstractcore
```

```bash
abstractcore serve
```

--------------------------------

### Install and run the AbstractCore server

Source: https://github.com/lpalbou/abstractcore/blob/main/README.md

Install the abstractcore package and start the server. The server prints a one-time console link and runs on 127.0.0.1:8000. On a network, set ABSTRACTCORE_AUTH_TOKEN yourself and pass --host 0.0.0.0.

```bash
pip install abstractcore                  # the server is part of the light install
abstractcore serve                        # 127.0.0.1:8000; prints a one-time console link
export ABSTRACTCORE_AUTH_TOKEN="$(abstractcore serve --print-token)"
```

--------------------------------

### Test Setup with OpenAI-Compatible Provider

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Basic generation using the openai-compatible provider.

```python
from abstractcore import create_llm

llm = create_llm("openai-compatible", model="default", base_url="http://localhost:1234/v1")
resp = llm.generate("Say hello in French")
print(resp.content)
```

--------------------------------

### Guided Decoding with Regex and JSON

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Syntax-constrained generation using guided_regex and guided_json.

```python
# Regex-constrained generation
response = llm.generate(
    "Write a Python function",
    guided_regex=r"def \w+\([^)]*\):\n(?:\s{4}.*\n)+"
)

# JSON schema enforcement
response = llm.generate(
    "Extract person info",
    guided_json={"type": "object", "properties": {...}}
)
```

--------------------------------

### Install abstractcore and start the HTTP server

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/faq.md

Install the light package and run the optional HTTP server. The server exposes an OpenAI-compatible /v1 endpoint and prints a link to the web console.

```bash
pip install abstractcore
abstractcore serve        # open the printed link for the web console
```

--------------------------------

### Install and run abstractcore-console

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/console-tui.md

Install the CLI and the console, then launch the console. The console reads the config file directly and runs the abstractcore CLI for derived or written content. Use --wizard for the guided setup.

```bash
pip install abstractcore               # the CLI the console drives
cargo install abstractcore-console     # the console (Rust 1.87+)
abstractcore-console                   # --wizard for the guided setup
```

--------------------------------

### Install Ollama on Linux

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Install Ollama on Linux using the official install script.

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

--------------------------------

### Run the Test Script

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Executes the test_setup.py script to run the provider tests.

```bash
python test_setup.py
```

--------------------------------

### Start Ollama and manage models

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Ensure Ollama is running, list available models, and pull a model if none are available.

```bash
# Make sure Ollama is running
ollama serve

# Check if models are available
ollama list

# Pull a model if none available
ollama pull gemma3:1b
```

--------------------------------

### Default Model Setup with Ollama

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/apps/basic-summarizer.md

Setup instructions for the default model gemma3:1b-it-qat. Requires installing Ollama and pulling the model before using the summarizer directly.

```bash
# Install Ollama: https://ollama.com/
# Download the default model
ollama pull gemma3:1b-it-qat

# Then use directly
summarizer document.txt
```

--------------------------------

### Install AbstractCore from source

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/agent-usage.md

Install the version of AbstractCore that matches this guide's source. Use the commented alternatives for local Apple or GPU engines.

```bash
python -m pip install -e .                 # run at the AbstractCore repository root
# For local engines instead: python -m pip install -e ".[apple]" or -e ".[gpu]"
```

--------------------------------

### Test LMStudio Setup

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Test the LMStudio provider by generating a response.

```python
from abstractcore import create_llm

# LM Studio exposes an OpenAI-compatible server (default: http://localhost:1234/v1).
# Use the model ID shown in LM Studio (or try "local-model" if unsure).
llm = create_llm("lmstudio", model="local-model", base_url="http://localhost:1234/v1")
resp = llm.generate("Hello, how are you?")
print(resp.content)
```

--------------------------------

### Test Portkey setup with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Verifies the Portkey provider setup by creating an LLM instance with a config_id and generating a response. Uses the 'gpt-4o-mini' model.

```python
from abstractcore import create_llm

llm = create_llm("portkey", model="gpt-4o-mini", config_id="pcfg_...")
resp = llm.generate("Say hello in French")
print(resp.content)
```

--------------------------------

### AbstractCore CLI configuration commands

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/centralized-config.md

Interactive guided setup with 8 steps (model, vision, API keys, server auth, audio, video, embeddings, logging). --install checks readiness and downloads what's missing; add --yes for non-interactive auto-download. --status displays current configuration.

```bash
# Interactive guided setup (8 steps: model, vision, API keys, server auth, audio, video, embeddings, logging)
abstractcore --config

# Check readiness of all subsystems + download/install what's missing
abstractcore --install

# Auto-download everything missing (non-interactive)
abstractcore --install --yes

# View current configuration
abstractcore --status
```

--------------------------------

### Start vLLM server for single GPU (7B-14B models)

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

For a single GPU with 24GB+ VRAM, works for 7B-14B models. Serves the model on port 8000.

```bash
# Single GPU (24GB+) - Works for 7B-14B models
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct --port 8000
```

--------------------------------

### Download MLX Models

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Download an MLX model using the abstractcore CLI.

```bash
abstractcore models download mlx mlx-community/Qwen3-4B-4bit
```

--------------------------------

### create_llm() usage examples

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/api-reference.md

Shows basic usage, configuration with temperature and max_tokens, and local provider setup for create_llm.

```python
from abstractcore import create_llm

# Basic usage
llm = create_llm("openai", model="gpt-4o-mini")

# With configuration
llm = create_llm(
    "anthropic",
    model="claude-haiku-4-5",
    temperature=0.7,
    max_tokens=1000,
    timeout=30
)

# Local provider
llm = create_llm("ollama", model="qwen2.5-coder:7b", base_url="http://localhost:11434")
```

--------------------------------

### Remote OpenAI example with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/README.md

Works with the light install; set `OPENAI_API_KEY` environment variable. Uses the OpenAI provider with a remote model.

```python
from abstractcore import create_llm

llm = create_llm("openai", model="gpt-4o-mini")
response = llm.generate("What is the capital of France?")
print(response.content)
```

--------------------------------

### Test OpenAI setup with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Verifies the OpenAI provider setup by creating an LLM instance and generating a response. Uses the 'gpt-4o-mini' model and expects the output 'Bonjour!'.

```python
from abstractcore import create_llm

# Test with an example model (use any model available on your account)
llm = create_llm("openai", model="gpt-4o-mini")
response = llm.generate("Say hello in French")
print(response.content)  # Should output: "Bonjour!"
```

--------------------------------

### Test HuggingFace setup with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Verifies the HuggingFace provider works. Uses a small model that auto-downloads. The model 'microsoft/DialoGPT-medium' is good for conversation.

```python
from abstractcore import create_llm

# Use a small model for testing (auto-downloads)
llm = create_llm("huggingface", model="microsoft/DialoGPT-medium")
response = llm.generate("Hello there!")
print(response.content)
```

--------------------------------

### Test MLX Setup

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Test the MLX provider by generating a response.

```python
from abstractcore import create_llm

# Test with a model you already downloaded/prefetched
llm = create_llm("mlx", model="mlx-community/Qwen3-4B-4bit")
response = llm.generate("Explain machine learning briefly")
print(response.content)
```

--------------------------------

### Installation commands for abstractcore

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/embeddings.md

Install the light version for remote providers, or use extras for local engines on Apple silicon or NVIDIA/AMD.

```bash
pip install abstractcore
```

```bash
pip install "abstractcore[apple]"   # Apple silicon
```

```bash
pip install "abstractcore[gpu]"     # NVIDIA / AMD
```

--------------------------------

### Install Ollama and download model

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/apps/basic-extractor.md

Commands to install Ollama and download the qwen3 model. Run 'ollama serve' to start the service. Used when encountering the 'Failed to initialize default Ollama model' error.

```bash
# Install Ollama and download model
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama serve
```

--------------------------------

### CLI configuration and interactive chat

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/getting-started.md

Optional CLI commands: configure defaults and API keys with 'abstractcore --config', check status with '--status', and start an interactive chat with 'abstractcore-chat' specifying provider and model.

```bash
# Configure defaults and API keys
abstractcore --config
abstractcore --status

# Interactive chat
abstractcore-chat --provider openai --model gpt-4o-mini
```

--------------------------------

### Install Ollama on macOS

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Install Ollama on macOS using the official install script or by downloading from the website.

```bash
curl -fsSL https://ollama.com/install.sh | sh
# OR download from https://ollama.com/download
```

--------------------------------

### Install abstractcore with optional extras

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/apps/basic-extractor.md

Install the abstractcore package. The light install works with the default Ollama path and all cloud providers. Optional extras enable in-process local engines: 'apple' for Apple silicon (MLX, HF/GGUF, local media engines) and 'gpu' for NVIDIA/AMD (vLLM, HF/GGUF, local media engines). The default model requires Ollama; see the comments for setup steps.

```bash
# Light install: the default Ollama path and every cloud provider work with it.
pip install abstractcore

# Or, for in-process local engines:
pip install "abstractcore[apple]"    # Apple silicon: MLX, HF/GGUF, local media engines
pip install "abstractcore[gpu]"      # NVIDIA / AMD: vLLM, HF/GGUF, local media engines

# Default model requires Ollama (free, runs locally)
# 1. Install Ollama: https://ollama.com/
# 2. Download model: ollama pull qwen3:4b-instruct-2507-q4_K_M
# 3. Start Ollama service

# Alternative: use a cloud provider (set its API key; no extra install)
```

--------------------------------

### GET /acore/jobs/{id}

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/engines.md

Retrieves the status and details of a job, such as an engine installation job. Use this to follow the progress of an install started via POST /acore/engines/{id}/install.

```APIDOC
## GET /acore/jobs/{id}

### Description
Retrieves the status and details of a job, such as an engine installation job. Use this to follow the progress of an install started via POST /acore/engines/{id}/install.

### Method
GET

### Endpoint
/acore/jobs/{id}

### Parameters
#### Path Parameters
- **id** (string) - Required - The job identifier.

### Response
#### Success Response (200)
Returns the job details, including status and any result fields (e.g., `installed_after`, `version_after`).
```

--------------------------------

### Create Clean Environment and Fresh Install

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/troubleshooting.md

Set up a virtual environment and install AbstractCore with the desired extras to resolve dependency conflicts.

```bash
# Create clean environment
python3 -m venv .venv
source .venv/bin/activate  # Linux/Mac
# OR
.venv\Scripts\activate  # Windows

# Fresh install
pip install --upgrade pip
pip install abstractcore                  # light (remote providers)
# or: pip install "abstractcore[apple]"   # Apple silicon local engines
# or: pip install "abstractcore[gpu]"     # NVIDIA / AMD local engines
```

--------------------------------

### Check GPU setup with nvidia-smi

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Run before starting vLLM to avoid Out Of Memory (OOM) errors. Shows GPU name, VRAM capacity, and current usage.

```bash
# Check available GPUs
nvidia-smi

# Shows: GPU name, VRAM capacity, and current usage
# Example: 4x NVIDIA L4 (23GB each) = 92GB total
```

--------------------------------

### Open the console with abstractcore serve

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/console.md

Install abstractcore and start the server, then open the printed link to access the console. The link is valid for 10 minutes and one use; use `abstractcore serve --claim-url --port 8000` to print a fresh one.

```bash
pip install abstractcore
abstractcore serve
# then open the printed link: http://127.0.0.1:8000/console#claim=...
```

--------------------------------

### Install AbstractCore with vLLM support

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Installs the 'gpu' extra which includes vLLM. Requires NVIDIA CUDA 12.1+; not compatible with Apple Silicon, AMD GPUs, or CPU-only systems.

```bash
# Install AbstractCore with vLLM support
pip install "abstractcore[gpu]"

# This installs vLLM which requires NVIDIA CUDA
# If you get CUDA errors, ensure CUDA 12.1+ is installed:
# https://developer.nvidia.com/cuda-downloads
```

--------------------------------

### Async usage with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/getting-started.md

Shows how to use the async API: create an LLM instance with the 'openai' provider and model 'gpt-4o-mini', then call agenerate() to get a response. The example prints the response content and runs the main coroutine with asyncio.run().

```python
import asyncio
from abstractcore import create_llm

async def main():
    llm = create_llm("openai", model="gpt-4o-mini")
    resp = await llm.agenerate("Give me 3 bullet points about HTTP caching.")
    print(resp.content)

asyncio.run(main())
```

--------------------------------

### Install and run AbstractCore server locally

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/server.md

Installs the package and starts the server on the default loopback address. The server prints its config directory, URL, and a one-time console link valid for 10 minutes. The command also exports the server token for API clients and checks health with curl.

```bash
pip install abstractcore
abstractcore serve
# AbstractCore server
#   Config dir: ~/.abstractcore/config
#   URL:        http://127.0.0.1:8000
#   ...
#   Open the console (one-time link, valid 10 minutes):
#     http://127.0.0.1:8000/console#claim=...

export ABSTRACTCORE_AUTH_TOKEN="$(abstractcore serve --print-token)"   # for API clients
curl http://localhost:8000/health
# Response: {"status":"healthy"}
```

--------------------------------

### Universal Test Script for AbstractCore Providers

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Save as test_setup.py and run to test all providers. Requires API keys for cloud providers (OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY) and local providers (Ollama, MLX, LMStudio). The script skips providers without keys or when local servers are not running.

```python
#!/usr/bin/env python3
"""Test script for AbstractCore providers"""

import os
from abstractcore import create_llm

def test_provider(provider_name, model, **kwargs):
    """Test a specific provider"""
    try:
        print(f"\n🧪 Testing {provider_name} with {model}...")
        llm = create_llm(provider_name, model=model, **kwargs)
        response = llm.generate("Say 'Hello from AbstractCore!'")
        print(f"[OK] {provider_name}: {response.content}")
        return True
    except Exception as e:
        print(f"[FAIL] {provider_name}: {e}")
        return False

def main():
    print("AbstractCore Provider Test Suite")
    print("=" * 40)

    results = {}

    # Test cloud providers (if API keys available)
    if os.getenv("OPENAI_API_KEY"):
        results["OpenAI"] = test_provider("openai", "gpt-4o-mini")
    else:
        print("\n⚠️  Skipping OpenAI (no OPENAI_API_KEY)")

    if os.getenv("ANTHROPIC_API_KEY"):
        results["Anthropic"] = test_provider("anthropic", "claude-haiku-4-5")
    else:
        print("\n⚠️  Skipping Anthropic (no ANTHROPIC_API_KEY)")

    if os.getenv("OPENROUTER_API_KEY"):
        results["OpenRouter"] = test_provider("openrouter", "openai/gpt-4o-mini")
    else:
        print("\n⚠️  Skipping OpenRouter (no OPENROUTER_API_KEY)")

    # Test local providers
    results["Ollama"] = test_provider("ollama", "gemma3:1b")

    try:
        results["MLX"] = test_provider("mlx", "mlx-community/Llama-3.2-3B-Instruct-4bit")
    except:
        print("\n⚠️  Skipping MLX (not on Apple Silicon or model not available)")

    try:
        # Note: OpenAI-compatible servers expect `/v1` in the base URL (LM Studio default is http://localhost:1234/v1)
        results["LMStudio"] = test_provider("lmstudio", "qwen/qwen3-4b-2507", base_url="http://localhost:1234/v1")
    except:
        print("\n⚠️  Skipping LMStudio (server not running on localhost:1234)")

    # Summary
    print("\n" + "=" * 40)
    print("Test Results:")
    working = [name for name, success in results.items() if success]
    if working:
        print(f"[OK] Working providers: {', '.join(working)}")
    else:
        print("[FAIL] No providers working")

    print("\n[INFO] Next steps:")
    print("- Add API keys for cloud providers")
    print("- Install Ollama and download models")
    print("- Start LMStudio local server")
    print("- See docs/prerequisites.md for detailed setup")

if __name__ == "__main__":
    main()
```

--------------------------------

### create_llm factory usage examples

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/architecture.md

Python examples showing how to instantiate providers via create_llm. Gateway providers (OpenRouter/Portkey) require API keys and route to external backends; AbstractCore forwards only explicit generation parameters.

```python
from abstractcore import create_llm

# Factory creates the right provider with proper configuration
llm = create_llm("openai", model="gpt-4o-mini", temperature=0.7)

# OpenAI-compatible /v1 endpoints (LMStudio, vLLM, custom proxies)
llm_local = create_llm("lmstudio", model="qwen/qwen3-4b-2507", base_url="http://localhost:1234/v1")
llm_openrouter = create_llm("openrouter", model="openai/gpt-4o-mini")  # requires OPENROUTER_API_KEY
llm_portkey = create_llm("portkey", model="gpt-4o-mini", config_id="pcfg_...")  # requires PORTKEY_API_KEY + PORTKEY_CONFIG
```

--------------------------------

### Install light setting from PyPI

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/server.md

The image is built from PyPI, not from the repository checkout, and installs the light setting.

```bash
abstractcore==<version>
```

--------------------------------

### Dry-run model download and engine install commands

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/agent-usage.md

Plans a download with explicit engine and artifact using the example selector `unsloth/Qwen3-8B-GGUF:Q4_K_M`. Also dry-runs an engine install. A plan is not installation; drop `--dry-run` to execute. Downloads may start jobs; use `abstractcore models jobs --json` and inspect final status before loading.

```bash
abstractcore models download huggingface "unsloth/Qwen3-8B-GGUF:Q4_K_M" --dry-run --json
abstractcore engines install ollama --dry-run
```

--------------------------------

### Catalog, fit, install, and engine inventory usage

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/api.md

Backs the `abstractcore models ...`, `abstractcore engines ...` commands and `/acore/models|engines|jobs` routes. Each function returns a JSON-ready dict with a `schema` field. `catalog` filters by model name and optional fit, `list_installed` lists installed artifacts for an engine, `host_jobs.start_download_job` launches a background download, and `engine_install_plan` returns the exact install command.

```python
from abstractcore.utils.host_profile import host_profile          # host_profile_v1
from abstractcore.utils.model_fit import estimate_fit             # fit verdict for one artifact
from abstractcore.config.model_catalog import catalog             # model_catalog_v1
from abstractcore.config.model_materializer import list_installed, delete_artifact
from abstractcore.config.engines import engine_inventory, engine_install_plan, engine_install
from abstractcore.config import host_jobs                         # host_job_v1 registry

rows = catalog("qwen3 8b", fits=True)["rows"]
installed = list_installed("ollama")["rows"]
job = host_jobs.start_download_job("ollama", "qwen3:8b")          # background job
print(host_jobs.default_registry().wait(job["job_id"])["status"])
print(engine_install_plan("ollama")["argv"])                       # the exact command
```

--------------------------------

### Start vLLM server with tensor parallelism for multiple GPUs

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Recommended for 30B models on multiple GPUs (e.g., 4x NVIDIA L4). Uses --tensor-parallel-size 4 and increased memory utilization.

```bash
# Multiple GPUs (RECOMMENDED for 30B models) - Use tensor parallelism
# Example: 4x NVIDIA L4 (23GB each)
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --host 0.0.0.0 --port 8000 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 8192 \
    --max-num-seqs 128
```

--------------------------------

### Start vLLM server for 30B models on single GPU

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

For 30B models on a single GPU (24GB+), reduce memory usage with --gpu-memory-utilization 0.85 and --max-model-len 4096 to avoid OOM.

```bash
# Single GPU (24GB+) - For 30B models, reduce memory
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --port 8000 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096
```

--------------------------------

### Startup options for abstractcore serve and compatibility entrypoints

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/server.md

Start the server using the built-in CLI, compatibility module entrypoints, or uvicorn directly. Includes options for debug, custom ports, host binding, token printing, and development auto-reload.

```bash
# Using AbstractCore's built-in CLI
abstractcore serve --help                              # View all options
abstractcore serve --debug                             # Debug mode
abstractcore serve --port 8080                         # Custom port (binds 127.0.0.1)
abstractcore serve --host 0.0.0.0 --port 8080          # Every interface (set ABSTRACTCORE_AUTH_TOKEN)
abstractcore serve --print-token                       # Print the local server's token, then exit
abstractcore serve --claim-url --port 8080             # Print a fresh one-time console link, then exit
abstractcore serve --debug --port 8001                 # Debug on custom port
abstractcore serve --reload                            # Development auto-reload

# Compatibility module entrypoints
python -m abstractcore.server
python -m abstractcore.server.app

# Using uvicorn directly
uvicorn abstractcore.server.app:app --reload                # Development with auto-reload
uvicorn abstractcore.server.app:app --workers 4             # Production with multiple workers
uvicorn abstractcore.server.app:app --port 3000             # Custom port
```

--------------------------------

### Set Anthropic API key environment variable

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Configure the ANTHROPIC_API_KEY environment variable for Anthropic. Provides three options: temporary export, permanent shell profile addition, or a project .env file. The key starts with 'sk-ant-'.

```bash
# Option 1: Export in terminal (temporary)
export ANTHROPIC_API_KEY="sk-ant-your-actual-api-key-here"

# Option 2: Add to shell profile (permanent)
echo 'export ANTHROPIC_API_KEY="sk-ant-your-actual-api-key-here"' >> ~/.bashrc
source ~/.bashrc

# Option 3: Create .env file
echo 'ANTHROPIC_API_KEY=sk-ant-your-actual-api-key-here' > .env
```

--------------------------------

### Set OpenAI API key environment variable

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Configure the OPENAI_API_KEY environment variable for OpenAI. Provides three options: temporary export, permanent shell profile addition, or a project .env file. The key starts with 'sk-'.

```bash
# Option 1: Export in terminal (temporary)
export OPENAI_API_KEY="sk-your-actual-api-key-here"

# Option 2: Add to ~/.bashrc or ~/.zshrc (permanent)
echo 'export OPENAI_API_KEY="sk-your-actual-api-key-here"' >> ~/.bashrc
source ~/.bashrc

# Option 3: Create .env file in your project
echo 'OPENAI_API_KEY=sk-your-actual-api-key-here' > .env
```

--------------------------------

### Run abstractcore models recommendations --host

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/recommended-models.md

Run this command to get fit verdicts for your own machine.

```bash
abstractcore models recommendations --host
```

--------------------------------

### Ollama Provider: Use Ollama embeddings

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/embeddings.md

Requires Ollama running locally. Setup includes installing Ollama and pulling an embedding model, e.g., granite-embedding:278m. Other popular models include nomic-embed-text (274MB) and granite-embedding:107m (smaller, faster).

```python
# Setup: Install Ollama and pull an embedding model
# ollama pull granite-embedding:278m

# Use Ollama embeddings
embedder = EmbeddingManager(
    provider="ollama",
    model="granite-embedding:278m"
)

# Other popular Ollama embedding models:
# - nomic-embed-text (274MB)
# - granite-embedding:107m (smaller, faster)
```

--------------------------------

### Test Ollama Setup

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Test the Ollama provider by generating a response.

```python
from abstractcore import create_llm

# Test with any model you installed via `ollama pull ...`
llm = create_llm("ollama", model="qwen3:4b-instruct-2507-q4_K_M")
response = llm.generate("What is Python?")
print(response.content)
```

--------------------------------

### /system command usage and output examples

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/acore-cli.md

Shows how to display or change the system prompt. /system shows the current prompt; /system [prompt] updates it. Output examples for both actions are included.

```bash
# Show current system prompt
/system

# Change system prompt
/system You are a Python expert focused on data science and machine learning.

# Output example for show:
[SYS] Current System Prompt:
==================================================
📝 Fixed Part:
You are a helpful AI assistant.

🔧 Full Prompt (as seen by LLM):
System Message 1 (Base):
You are a helpful AI assistant.
==================================================

# Output example for change:
[OK] System prompt updated!
📝 Old: You are a helpful AI assistant.
📝 New: You are a Python expert focused on data science and machine learning.
```

--------------------------------

### Initial Setup Workflow

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/centralized-config.md

Complete initial setup: check status, set global and app-specific defaults, configure API keys, optional server gateway, logging, streaming, and verify. Includes optional OpenRouter and OpenAI-compatible keys.

```bash
# 1. Check current status
abstractcore --status

# 2. Set global fallback
abstractcore --set-global-default ollama/llama3:8b

# 3. Configure specific apps for optimal performance
abstractcore --set-app-default summarizer openai gpt-4o-mini
abstractcore --set-app-default extractor ollama qwen3:4b-instruct
abstractcore --set-app-default judge anthropic claude-haiku-4-5

# 4. Set API keys as needed
abstractcore --set-api-key openai sk-your-key-here
abstractcore --set-api-key anthropic your-anthropic-key
# Optional (only if you plan to use the OpenRouter provider):
abstractcore --set-api-key openrouter your-openrouter-key
abstractcore --set-api-key openai-compatible your-endpoint-key
abstractcore --set-api-key vllm your-vllm-key

# 5. Optional: configure the OpenAI-compatible HTTP server gateway
abstractcore --set-server-auth-token acore-server-secret
abstractcore --set-server-base-url-allowlist "https://example.com/v1"

# 6. Configure logging for development
abstractcore --enable-debug-logging
abstractcore --enable-file-logging

# 7. Enable streaming for interactive CLI
abstractcore --stream on

# 8. Verify configuration
abstractcore --status
```

--------------------------------

### Install AbstractCore (light, apple, gpu)

Source: https://github.com/lpalbou/abstractcore/blob/main/README.md

Three pip install options: light for remote providers, apple for Mac local engines, gpu for NVIDIA/AMD local engines. Apple and GPU include everything in light.

```bash
# Light: every remote provider (OpenAI, Anthropic, OpenRouter, Portkey, any OpenAI-compatible
# endpoint, LM Studio, Ollama, a vLLM server), tools, media inputs, the HTTP server,
# and the voice / vision / music / 3D capability plugins
pip install abstractcore
```

```bash
# Apple silicon: light + every local engine a Mac can run (MLX text and vision,
# HuggingFace/GGUF, local embeddings, local voice, image, video and music engines)
pip install "abstractcore[apple]"
```

```bash
# NVIDIA / AMD GPU: light + every local engine a GPU machine can run (vLLM,
# HuggingFace/GGUF, local embeddings, local voice, image, video and music engines)
pip install "abstractcore[gpu]"
```

--------------------------------

### POST /acore/jobs/{id}/cancel

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/engines.md

Cancels a running job, such as an engine installation. This stops the installer and every process it started.

```APIDOC
## POST /acore/jobs/{id}/cancel

### Description
Cancels a running job, such as an engine installation. This stops the installer and every process it started.

### Method
POST

### Endpoint
/acore/jobs/{id}/cancel

### Parameters
#### Path Parameters
- **id** (string) - Required - The job identifier.

### Response
#### Success Response (200)
Confirms the cancellation.
```

--------------------------------

### Test OpenRouter setup with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Verifies the OpenRouter provider setup by creating an LLM instance and generating a response. Uses the 'openai/gpt-4o-mini' model.

```python
from abstractcore import create_llm

llm = create_llm("openrouter", model="openai/gpt-4o-mini")
resp = llm.generate("Say hello in French")
print(resp.content)
```

--------------------------------

### Test Anthropic setup with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/prerequisites.md

Verifies the Anthropic provider setup by creating an LLM instance and generating a response. Uses the 'claude-haiku-4-5' model.

```python
from abstractcore import create_llm

# Test with an example model (use any model available on your account)
llm = create_llm("anthropic", model="claude-haiku-4-5")
response = llm.generate("Explain Python in one sentence")
print(response.content)
```

--------------------------------

### Apply music recommendation with abstractcore config set-default and models download

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/recommended-models.md

Music is an opt-in recommendation. Use these commands to set the default output.music provider and model, then download the model from the local Hugging Face cache. AbstractMusic loads ACE-Step only from the local cache, so download it first.

```bash
abstractcore config set-default output.music --provider acestep --model ACE-Step/acestep-v15-xl-turbo-diffusers
abstractcore models download diffusers ACE-Step/acestep-v15-xl-turbo-diffusers
```

--------------------------------

### Calculator Tool with create_llm and tool_calls

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/examples.md

Defines calculate and sqrt tools for mathematical operations, then uses create_llm with the 'openai' provider to generate a response. The response includes tool_calls that the host/runtime executes. The tool definitions specify parameters and required fields.

```python
from abstractcore import create_llm
import math

def calculate(expression: str) -> str:
    """Safely evaluate mathematical expressions."""
    try:
        # In production, use a proper expression parser
        # This is simplified for demo purposes
        allowed_chars = set('0123456789+-*/.() ')
        if not all(c in allowed_chars for c in expression):
            return "Error: Invalid characters in expression"

        result = eval(expression)
        return f"{expression} = {result}"
    except Exception as e:
        return f"Error calculating {expression}: {str(e)}"

def sqrt(number: float) -> str:
    """Calculate square root."""
    try:
        result = math.sqrt(number)
        return f"√{number} = {result}"
    except Exception as e:
        return f"Error: {str(e)}"

# Tool definitions
tools = [
    {
        "name": "calculate",
        "description": "Perform basic mathematical calculations",
        "parameters": {
            "type": "object",
            "properties": {
                "expression": {"type": "string", "description": "Mathematical expression"}
            },
            "required": ["expression"]
        }
    },
    {
        "name": "sqrt",
        "description": "Calculate square root of a number",
        "parameters": {
            "type": "object",
            "properties": {
                "number": {"type": "number", "description": "Number to calculate square root of"}
            },
            "required": ["number"]
        }
    }
]

llm = create_llm("openai", model="gpt-4o-mini")

response = llm.generate(
    "What is 25 * 4 + 12, and what's the square root of 144?",
    tools=tools
)

print(response.content)
print(response.tool_calls)  # Structured tool call requests (host/runtime executes them)
```

--------------------------------

### Get application identity and about lines

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/api.md

Use app_identity() to get identity facts for an installed application, then render them as lines with about_lines(). gateway_version_rows() converts a gateway about payload into rows. app_identity() raises KeyError for unknown app ids and PackageNotFoundError when no version is given and the distribution is not installed.

```python
from abstractcore.utils.identity import about_lines, app_identity, gateway_version_rows

ident = app_identity("abstractcore")          # version defaults to the installed distribution
print("\n".join(about_lines(ident)))
rows = gateway_version_rows({"abstractgateway": "0.4.2", "abstractframework": "0.3.0", "packages": {}})
```

--------------------------------

### POST /v1/camera/open

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/server.md

Claims a camera and starts its session.

```APIDOC
## [POST] /v1/camera/open

### Description
Claim a camera and start its session.

### Method
POST

### Endpoint
`/v1/camera/open`

### Parameters
#### Request Body
- **camera** (string) - Optional - Camera identifier
```

--------------------------------

### Quick CLI Usage Examples

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/apps/basic-summarizer.md

Basic command-line usage for the summarizer CLI. Shows simple usage, style and length options, focus areas, and provider/model selection. PDF and Office inputs work with the light install.

```bash
# Simple usage (PDF and Office inputs work with the light install)
summarizer document.pdf

# With specific style and length
summarizer report.txt --style executive --length brief

# Focus on specific aspects
summarizer data.md --focus "technical details" --output summary.txt

# Use different provider
summarizer large.txt --provider openai --model gpt-4o-mini --verbose
```

--------------------------------

### GET /acore/models/installed

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/server.md

Get installed models per engine with sizes and delete blockers.

```APIDOC
## [GET] /acore/models/installed

### Description
Get installed models per engine with sizes and delete blockers.

### Method
GET

### Endpoint
/acore/models/installed

### Parameters
#### Query Parameters
- `provider` (string) - Optional - Provider filter

### Response
#### Success Response (200)
- No specific response fields documented
```

--------------------------------

### Simple Q&A with create_llm

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/examples.md

Basic usage of AbstractCore's create_llm to generate a response. Works with any provider by changing the provider name.

```python
from abstractcore import create_llm

# Works with any provider
llm = create_llm("openai", model="gpt-4o-mini")  # or "anthropic", "ollama"...

response = llm.generate("What is the difference between Python and JavaScript?")
print(response.content)
```

--------------------------------

### GET /acore/models/installed

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/models.md

Lists installed models, optionally filtered by provider. Returns a list of installed model artifacts.

```APIDOC
## [GET] /acore/models/installed

### Description
Lists installed models, optionally filtered by provider. Returns a list of installed model artifacts.

### Method
GET

### Endpoint
/acore/models/installed

### Parameters
#### Query Parameters
- **provider** (string) - Optional - Filter by provider (e.g., ollama, lmstudio).

### Response
#### Success Response (200)
Returns a JSON array of installed model entries.
```

--------------------------------

### MLX/HF async example with fallback

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/async-guide.md

Shows how to use MLX or HuggingFace providers with asyncio.to_thread() fallback to keep the event loop responsive. Mix inference with async I/O using asyncio.gather().

```python
# MLX/HF async example (fallback keeps event loop responsive)
import asyncio
from abstractcore import create_llm

async def main():
    llm = create_llm("mlx", model="mlx-community/Qwen3-4B-4bit")

    # Can mix MLX inference with async I/O
    inference_task = llm.agenerate("What is Python?")
    io_task = fetch_data_from_api()  # Async I/O

    # Both run concurrently - event loop not blocked!
    response, data = await asyncio.gather(inference_task, io_task)

asyncio.run(main())
```

--------------------------------

### Install engine via HTTP API

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/engines.md

HTTP POST request to install an engine, returning a job to follow with GET /acore/jobs/{id}. A second install request is refused with exit 2 and HTTP 409.

```http
POST /acore/engines/{id}/install
```

--------------------------------

### Run Async CLI Demo with LM Studio Provider and Streaming

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/acore-cli.md

Educational async CLI demo for learning async patterns in AbstractCore. Use with the lmstudio provider and qwen/qwen3-vl-30b model, with streaming enabled.

```bash
python examples/cli/async_cli_demo.py --provider lmstudio --model qwen/qwen3-vl-30b --stream
```

--------------------------------

### Models Not Showing: Check Providers and Provider Setup

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/troubleshooting.md

Commands to check provider configuration and verify each provider (Ollama, OpenAI, Anthropic, LMStudio) is set up correctly. Use when the models list is empty.

```bash
curl http://localhost:8000/providers
```

```bash
ollama list  # Should show models
ollama serve  # Make sure it's running
```

```bash
echo $OPENAI_API_KEY  # Should be set
```

```bash
echo $ANTHROPIC_API_KEY  # Should be set
```

```bash
curl http://localhost:1234/v1/models  # Should return models
```

--------------------------------

### GET /acore/engines

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/engines.md

Retrieves the list of engines and reports the effective install_allowed setting. This endpoint is used to check whether engine installation is permitted on the server.

```APIDOC
## [GET] /acore/engines

### Description
Retrieves the list of engines and reports the effective `install_allowed` setting.

### Method
GET

### Endpoint
/acore/engines

### Response
#### Success Response (200)
- **install_allowed** (boolean) - Indicates whether engine installation is allowed.

```

--------------------------------

### Run Async CLI Demo with Ollama Provider

Source: https://github.com/lpalbou/abstractcore/blob/main/docs/acore-cli.md

Educational async CLI demo for learning async patterns in AbstractCore. Use with the ollama provider and qwen3:4b model.

```bash
python examples/cli/async_cli_demo.py --provider ollama --model qwen3:4b
```
Tokens:
Raw