### Install OpenGauss from Repository

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/getting-started/installation.md

Executes the local installation script from the root of the checked-out repository. This script configures the environment, sets up runtime directories in ~/.gauss/, and links the binary to ~/.local/bin/gauss.

```bash
./scripts/install.sh
```

--------------------------------

### Verify Installation and Initialize Project

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/getting-started/installation.md

Commands to verify the installation by invoking the CLI and initializing a new project workspace. These commands are run after the installation script has completed successfully.

```bash
gauss
```

```text
/project init
```

--------------------------------

### Install and Setup TorchTitan

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/training/torchtitan/SKILL.md

Instructions for installing the TorchTitan library and downloading necessary Hugging Face assets like tokenizers for pretraining.

```bash
pip install torchtitan
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt
```

```bash
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=YOUR_HF_TOKEN
```

--------------------------------

### Dockerizing vLLM Application with Outlines Integration

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/references/backends.md

This Dockerfile demonstrates how to set up a production environment for serving LLM applications using vLLM and Outlines. It starts from a vLLM base image, installs the Outlines library, copies the application code, and defines the command to run the application, facilitating easy deployment and scaling.

```dockerfile
FROM vllm/vllm-openai:latest

# Install outlines
RUN pip install outlines

# Copy your code
COPY app.py /app/

# Run
CMD ["python", "/app/app.py"]
```

--------------------------------

### Install AutoGPTQ for Model Quantization

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/vllm/references/quantization.md

Provides the command to install the AutoGPTQ library, required for quantizing models using the GPTQ method. This installation step is necessary before you can use AutoGPTQ for custom model quantization.

```bash
# Install AutoGPTQ
pip install auto-gptq
```

--------------------------------

### Install Segment Anything Model from GitHub

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/segment-anything/references/troubleshooting.md

Provides commands to install the Segment Anything model directly from its GitHub repository, either as a package or by cloning and installing locally. Includes a verification step.

```bash
# Install from GitHub
pip install git+https://github.com/facebookresearch/segment-anything.git

# Or clone and install
git clone https://github.com/facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

# Verify installation
python -c "from segment_anything import sam_model_registry; print('OK')"
```

--------------------------------

### Configure Installation Options

Source: https://github.com/math-inc/opengauss/blob/main/README.md

Optional flags for the installation script to customize environment setup, such as pre-warming workspaces or skipping system package installation.

```bash
./scripts/install.sh --with-workspace
./scripts/install.sh --skip-system-packages
./scripts/install.sh --recreate-venv
```

--------------------------------

### Setup and Installation Commands for Gauss Agent

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/developer-guide/contributing.md

Commands to clone the repository, create a virtual environment, install dependencies, and configure the development environment.

```bash
git clone --recurse-submodules https://github.com/NousResearch/gauss-agent.git
cd gauss-agent
uv venv venv --python 3.11
export VIRTUAL_ENV="$(pwd)/venv"
uv pip install -e ".[all,dev]"
uv pip install -e "./mini-swe-agent"
uv pip install -e "./tinker-atropos"
npm install
```

--------------------------------

### Install and Verify AudioCraft Library

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/audiocraft/references/troubleshooting.md

Installs the AudioCraft library from PyPI or GitHub and provides a command to verify the installation by importing a model.

```bash
pip install audiocraft

# Or from GitHub
pip install git+https://github.com/facebookresearch/audiocraft.git

# Verify installation
python -c "from audiocraft.models import MusicGen; print('OK')"
```

--------------------------------

### Configure Gauss Agent Gateway

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/user-guide/messaging/discord.md

Commands to initialize the interactive setup wizard and start the Gauss Agent gateway service.

```bash
gauss gateway setup
gauss gateway
```

--------------------------------

### Installing Python Packages from Git

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/references/advanced-usage.md

Demonstrates how to install Python packages directly from a Git repository using `pip`. This is useful for using development versions or specific branches of libraries.

```python
image = modal.Image.debian_slim().pip_install(
    "git+https://github.com/huggingface/transformers.git@main"
)
```

--------------------------------

### Install AutoAWQ for Model Quantization

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/vllm/references/quantization.md

Provides the command to install the AutoAWQ library, which is necessary for quantizing models using the AWQ method. This is a prerequisite for running custom AWQ quantization scripts.

```bash
# Install AutoAWQ
pip install autoawq
```

--------------------------------

### Start llama-cpp Server (Bash)

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/llama-cpp/references/server.md

Basic command to start the llama.cpp server with a specified model, host, and port. Supports context size configuration. GPU acceleration can be enabled by offloading layers.

```bash
# Basic server
./llama-server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -c 4096  # Context size

# With GPU acceleration
./llama-server \
    -m models/llama-2-70b.Q4_K_M.gguf \
    -ngl 40  # Offload 40 layers to GPU
```

--------------------------------

### Launch GPTQ Quantized Model with vLLM

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/vllm/references/quantization.md

Shows how to serve a GPTQ quantized model using vLLM. GPTQ offers broad model support and good compression. This example includes basic launch command and options for activation ordering.

```bash
vllm serve TheBloke/Llama-2-13B-GPTQ \
  --quantization gptq \
  --dtype float16
```

```bash
# Specify GPTQ parameters if needed
vllm serve MODEL \
  --quantization gptq \
  --gptq-act-order \
  --dtype float16
```

--------------------------------

### Install Mod Loader

Source: https://github.com/math-inc/opengauss/blob/main/skills/gaming/minecraft-modpack-server/SKILL.md

Executes the mod loader installation script or installer jar to prepare the server environment without launching it.

```bash
cd ~/minecraft-server/server
ATM10_INSTALL_ONLY=true bash startserver.sh
# Or for generic Forge packs:
# java -jar forge-*-installer.jar --installServer
```

--------------------------------

### Configure Hugging Face Transformers Models

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/references/backends.md

Demonstrates how to initialize models using the Transformers backend, including basic setup, hardware-specific device configuration (CUDA, MPS, CPU), and advanced memory optimizations like quantization and multi-GPU distribution.

```python
import outlines

# Basic setup
model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")
generator = outlines.generate.json(model, YourModel)
result = generator("Your prompt")

# Device configuration
model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct", device="cuda")

# Advanced: Quantization and Multi-GPU
model = outlines.models.transformers(
    "meta-llama/Llama-3.1-70B-Instruct",
    device="cuda",
    model_kwargs={
        "load_in_4bit": True,
        "device_map": "auto"
    }
)
```

--------------------------------

### Install ACP Extra for Gauss Agent

Source: https://github.com/math-inc/opengauss/blob/main/docs/acp-setup.md

Installs the necessary ACP extra for Gauss Agent. This is a prerequisite for enabling ACP functionality. It uses pip to install the package in editable mode with the 'acp' extra.

```bash
pip install -e ".[acp]"
```

--------------------------------

### Run Hello World with GPU

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/SKILL.md

A basic example demonstrating how to define a Modal application and execute a function on a T4 GPU to retrieve system information.

```python
import modal

app = modal.App("hello-gpu")

@app.function(gpu="T4")
def gpu_info():
    import subprocess
    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout

@app.local_entrypoint()
def main():
    print(gpu_info.remote())
```

--------------------------------

### Install GroundingDINO and SAM for Text-Prompted Segmentation

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/segment-anything/references/advanced-usage.md

Installs the necessary libraries for Grounded SAM, which combines GroundingDINO for object detection with SAM for segmentation. This setup allows for text-based image segmentation.

```bash
pip install groundingdino-py
pip install git+https://github.com/facebookresearch/segment-anything.git
```

--------------------------------

### Configure and Start Gauss Agent Gateway (Shell)

Source: https://github.com/math-inc/opengauss/blob/main/landingpage/index.html

Commands for setting up and running the Gauss Agent messaging gateway. 'gauss gateway setup' launches an interactive wizard for gateway configuration, 'gauss gateway' starts the messaging gateway, and 'gauss gateway install' installs it as a systemd service for persistent operation.

```shell
# Interactive gateway setup wizard
gauss gateway setup

# Start the messaging gateway
gauss gateway

# Install as a system service
gauss gateway install
```

--------------------------------

### Reduce Modal Cold Start Times with Warm Containers and Caching

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/references/troubleshooting.md

Python examples for reducing Modal cold start times by keeping containers warm, loading models during container startup, and caching models in persistent volumes.

```python
# Keep containers warm
@app.function(
    container_idle_timeout=600,  # Keep warm 10 min
    keep_warm=1                  # Always keep 1 container ready
)
def low_latency():
    pass

# Load model during container start
@app.cls(gpu="A100")
class Model:
    @modal.enter()
    def load(self):
        # This runs once at container start, not per request
        self.model = load_heavy_model()

# Cache model in volume
volume = modal.Volume.from_name("models", create_if_missing=True)

@app.function(volumes={"/cache": volume})
def cached_model():
    if os.path.exists("/cache/model"):
        model = load_from_disk("/cache/model")
    else:
        model = download_model()
        save_to_disk(model, "/cache/model")
        volume.commit()
```

--------------------------------

### Full MCP Server Configuration Example (YAML)

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/user-guide/features/mcp.md

A comprehensive example showcasing the configuration of multiple MCP servers (github, stripe, legacy). It includes tool inclusion/exclusion, disabling utility wrappers, setting server URLs, and disabling a server entirely.

```yaml
mcp_servers:
  github:
    command: "npx"
    args: ["-y", "@modelcontextprotocol/server-github"]
    env:
      GITHUB_PERSONAL_ACCESS_TOKEN: "***"
    tools:
      include: [create_issue, list_issues, search_code]
      prompts: false

  stripe:
    url: "https://mcp.stripe.com"
    headers:
      Authorization: "Bearer ***"
    tools:
      exclude: [delete_customer]
      resources: false

  legacy:
    url: "https://mcp.legacy.internal"
    enabled: false
```

--------------------------------

### QMD CLI Commands for Querying and Management

Source: https://github.com/math-inc/opengauss/blob/main/optional-skills/research/qmd/SKILL.md

Demonstrates how to use the qmd CLI for querying information and managing collections when MCP is not configured. Includes examples for adding collections, contexts, embedding, and checking status.

```terminal
qmd query 'what was decided about the API redesign' --json
```

```terminal
qmd collection add ~/Documents/notes --name notes
```

```terminal
qmd context add qmd://notes 'Personal research notes and ideas'
```

```terminal
qmd embed
```

```terminal
qmd status
```

--------------------------------

### Generate API Key and Start TensorRT-LLM Server with Authentication

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/tensorrt-llm/references/serving.md

Provides bash commands to generate a secure API key using OpenSSL and then start the TensorRT-LLM server with this key for API authentication. It also shows an example of a client making an authenticated request.

```bash
# Generate API key
export API_KEY=$(openssl rand -hex 32)

# Start server with authentication
trtllm-serve meta-llama/Meta-Llama-3-8B \
    --api_key $API_KEY

# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "...", "messages": [...]}'
```

--------------------------------

### Basic Setup for llama.cpp Local Model (Python)

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/guidance/references/backends.md

Provides the basic setup for using the llama.cpp backend with the Guidance library. This allows running LLMs optimized for CPU and efficient memory usage. Requires the 'guidance' library and the llama.cpp bindings.

```python
from guidance.models import LlamaCpp

# Basic setup for llama.cpp model
# lm = LlamaCpp("path/to/your/model.gguf")
```

--------------------------------

### Custom Dockerfile Image Configuration

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/references/advanced-usage.md

Shows how to build a Modal image directly from a custom Dockerfile. This allows for complex environment setups and dependency installations not easily achievable with Modal's Python API.

```python
image = modal.Image.from_dockerfile("./Dockerfile")
```

--------------------------------

### Get Python Client Version

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/vector-databases/qdrant/references/troubleshooting.md

This bash command uses `pip show` to display detailed information about the installed `qdrant-client` package, including its version number. This information is necessary when reporting issues or seeking support.

```bash
pip show qdrant-client
```

--------------------------------

### Initial Sweep Configuration and Agent Launch (Python)

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/evaluation/weights-and-biases/references/sweeps.md

Demonstrates the initial setup for a Weights & Biases sweep using random search and launching an agent for a specified number of runs. This is a starting point for exploration before refining the search strategy.

```python
# Initial exploration: Random search, 20 runs
sweep_config_v1 = {
    'method': 'random',
    'parameters': {...}
}
wandb.agent(sweep_id_v1, train, count=20)

# Refined search: Bayes, narrow ranges
sweep_config_v2 = {
    'method': 'bayes',
    'parameters': {
        'learning_rate': {
            'min': 5e-5,  # Narrowed from 1e-6 to 1e-4
            'max': 1e-4
        }
    }
}
```

--------------------------------

### Initialize LlamaCpp Models

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/guidance/references/backends.md

Demonstrates basic loading of GGUF models with context window configuration and various GPU offloading strategies to balance performance and hardware utilization.

```python
# Load GGUF model
lm = LlamaCpp(
    model_path="/path/to/model.gguf",
    n_ctx=4096
)

# GPU acceleration with partial and full offload
lm = LlamaCpp(
    model_path="/path/to/model.gguf",
    n_ctx=4096,
    n_gpu_layers=35,
    n_threads=8
)

lm = LlamaCpp(
    model_path="/path/to/model.gguf",
    n_ctx=4096,
    n_gpu_layers=-1
)
```

--------------------------------

### vLLM Optimization: High Throughput and Multi-GPU Setup

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/references/backends.md

Illustrates how to configure vLLM for high-performance inference. The first example optimizes for high throughput by utilizing a large portion of GPU memory and increasing the maximum number of sequences processed concurrently, while disabling eager execution for performance. The second example shows how to set up vLLM for multi-GPU inference with a large model.

```python
model = outlines.models.vllm(
    "meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.95,  # Use 95% of GPU
    max_num_seqs=256,             # High concurrency
    enforce_eager=False           # Use CUDA graphs
)
```

```python
model = outlines.models.vllm(
    "meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,  # 4 GPUs
    gpu_memory_utilization=0.9
)
```

--------------------------------

### Full RL Training: Start Environment (Bash)

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/training/gauss-atropos-environments/references/usage-patterns.md

Launches the OpenGauss environment in 'serve' mode for actual RL training. This command requires a configuration file and is intended to be run in conjunction with the Atropos API server.

```bash
# Terminal 2: Start your environment
python environments/your_env.py serve \
  --config environments/your_env/default.yaml
```

--------------------------------

### Complete Git Workflow Example (Bash)

Source: https://github.com/math-inc/opengauss/blob/main/skills/github/github-pr-workflow/SKILL.md

A comprehensive bash script demonstrating a typical development workflow: starting from main, creating a new branch, making changes (placeholder), committing, pushing, creating a PR, monitoring CI, and merging.

```bash
# 1. Start from clean main
git checkout main && git pull origin main

# 2. Branch
git checkout -b fix/login-redirect-bug

# 3. (Agent makes code changes with file tools)

# 4. Commit
git add src/auth/login.py tests/test_login.py
git commit -m "fix: correct redirect URL after login

Preserves the ?next= parameter instead of always redirecting to /dashboard."

# 5. Push
git push -u origin HEAD

# 6. Create PR (picks gh or curl based on what's available)
# ... (see Section 3)

# 7. Monitor CI (see Section 4)

# 8. Merge when green (see Section 6)
```

--------------------------------

### Initialize GGUF Models with Outlines

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/references/backends.md

Demonstrates loading GGUF-formatted models with varying quantization levels (Q4 to F16) using the Outlines llamacpp backend. This is ideal for local inference on consumer hardware.

```python
import outlines

# Loading a Q4_K_M quantized model
model = outlines.models.llamacpp("./models/model.Q4_K_M.gguf")

# Loading a high-quality F16 model
model = outlines.models.llamacpp("./models/model.F16.gguf")
```

--------------------------------

### Configure LLM Backends

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/SKILL.md

Examples for initializing different model backends including Transformers, llama.cpp, and vLLM. Covers GPU offloading, quantization, and multi-GPU parallelization.

```python
# Transformers
model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct", device="cuda")

# llama.cpp
model = outlines.models.llamacpp("./models/llama-3.1-8b.Q4_K_M.gguf", n_gpu_layers=35)

# vLLM
model = outlines.models.vllm("meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=4)
```

--------------------------------

### Configure Documentation and Knowledge Servers

Source: https://github.com/math-inc/opengauss/blob/main/website/docs/guides/use-mcp-with-gauss.md

Demonstrates how to enable resource and prompt discovery for documentation-focused MCP servers. This allows the model to access shared knowledge assets rather than just executing direct actions.

```yaml
mcp_servers:
  docs:
    url: "https://mcp.docs.example.com"
    tools:
      prompts: true
      resources: true
```

--------------------------------

### Setup and Configuration for Jupyter Live Kernel

Source: https://github.com/math-inc/opengauss/blob/main/skills/data-science/jupyter-live-kernel/SKILL.md

Commands to initialize the environment, locate the script, and start a headless JupyterLab server for agent interaction.

```bash
SCRIPT="$HOME/.agent-skills/hamelnb/skills/jupyter-live-kernel/scripts/jupyter_live_kernel.py"
git clone https://github.com/hamelsmu/hamelnb.git ~/.agent-skills/hamelnb
uv run "$SCRIPT" servers
jupyter-lab --no-browser --port=8888 --notebook-dir=$HOME/notebooks --IdentityProvider.token='' --ServerApp.password='' > /tmp/jupyter.log 2>&1 &
```

--------------------------------

### Install CLIP and Dependencies

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/clip/SKILL.md

Installs the CLIP library from GitHub and essential dependencies like PyTorch, torchvision, ftfy, regex, and tqdm. These are required to run CLIP models.

```bash
pip install git+https://github.com/openai/CLIP.git
pip install torch torchvision ftfy regex tqdm
```

--------------------------------

### SLURM Environment Variable Detection in Lightning

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/training/pytorch-lightning/references/distributed.md

Lightning automatically detects and utilizes SLURM environment variables for distributed training configuration, simplifying setup for SLURM-managed clusters. This example shows how to initialize a Trainer with GPU acceleration and distributed data parallel strategy.

```python
import lightning as L

trainer = L.Trainer(
    accelerator='gpu',
    devices=8,
    num_nodes=4,  # From SBATCH --nodes
    strategy='ddp'
)
```

--------------------------------

### Install AudioCraft Library

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/audiocraft/SKILL.md

Installs the audiocraft library from PyPI or the latest version from GitHub. Alternatively, install dependencies for use with HuggingFace Transformers.

```bash
# From PyPI
pip install audiocraft

# From GitHub (latest)
pip install git+https://github.com/facebookresearch/audiocraft.git

# Or use HuggingFace Transformers
pip install transformers torch torchaudio
```

--------------------------------

### Install and Upgrade Modal Python Package

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/references/troubleshooting.md

Instructions for upgrading pip, installing the Modal package with a specific Python version, and installing from a wheel file.

```bash
# Upgrade pip
pip install --upgrade pip

# Install with specific Python version
python3.11 -m pip install modal

# Install from wheel
pip install modal --prefer-binary
```

--------------------------------

### Launch Distributed Training with Accelerate

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/training/accelerate/SKILL.md

Provides examples of how to launch a distributed training script using the `accelerate launch` command. This includes single GPU, multi-GPU, and multi-node configurations.

```bash
# Single GPU
accelerate launch train.py

# Multi-GPU (8 GPUs)
accelerate launch --multi_gpu --num_processes 8 train.py

# Multi-node
accelerate launch --multi_gpu --num_processes 16 \
  --num_machines 2 --machine_rank 0 \
  --main_process_ip $MASTER_ADDR \
  train.py
```

--------------------------------

### Python Bindings Installation

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/gguf/references/troubleshooting.md

Methods for installing llama-cpp-python with specific hardware support for CUDA or Metal acceleration.

```bash
pip install cmake scikit-build-core

# CUDA support
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

# Metal support
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --force-reinstall --no-cache-dir
```

--------------------------------

### Install FFmpeg for Audio Processing

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/audiocraft/references/troubleshooting.md

Provides commands to install FFmpeg on various operating systems (Ubuntu/Debian, macOS, Windows via conda) and a command to verify the FFmpeg installation.

```bash
# Ubuntu/Debian
sudo apt-get install ffmpeg

# macOS
brew install ffmpeg

# Windows (using conda)
conda install -c conda-forge ffmpeg

# Verify
ffmpeg -version
```

--------------------------------

### Initialize and Use Gauss Projects

Source: https://github.com/math-inc/opengauss/blob/main/README.md

Commands to create, initialize, or switch between Lean projects within the Gauss environment.

```bash
gauss
/project create ~/my-project
cd ~/my-lean-project
gauss
/project init
```

--------------------------------

### Install DDGS Package

Source: https://github.com/math-inc/opengauss/blob/main/skills/research/duckduckgo-search/SKILL.md

Installs the necessary Python package for using the DuckDuckGo search API. This is a one-time setup step.

```bash
# Install the ddgs package (one-time)
pip install ddgs
```

--------------------------------

### Launch Quantized Models with vLLM

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/vllm/SKILL.md

Demonstrates how to serve a pre-quantized model using the vLLM CLI, specifying quantization methods and GPU memory utilization settings.

```bash
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95
```

--------------------------------

### Install Modal CLI

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/SKILL.md

Instructions for installing the Modal Python package and authenticating the local environment with the cloud platform.

```bash
pip install modal
modal setup
```

--------------------------------

### Transformers Optimization: FP16, Flash Attention, and 8-bit Quantization

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/outlines/references/backends.md

Demonstrates how to optimize Transformers models for reduced memory usage and increased speed. It shows examples of using FP16 precision, enabling Flash Attention 2 for faster computation, and applying 8-bit quantization to decrease memory footprint. These optimizations are crucial for efficient deployment on limited hardware.

```python
model = outlines.models.transformers(
    "meta-llama/Llama-3.1-8B-Instruct",
    device="cuda",
    model_kwargs={"torch_dtype": "float16"}
)
```

```python
model = outlines.models.transformers(
    "meta-llama/Llama-3.1-8B-Instruct",
    device="cuda",
    model_kwargs={
        "torch_dtype": "float16",
        "use_flash_attention_2": True
    }
)
```

```python
model = outlines.models.transformers(
    "meta-llama/Llama-3.1-8B-Instruct",
    device="cuda",
    model_kwargs={
        "load_in_8bit": True,
        "device_map": "auto"
    }
)
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/lambda-labs/SKILL.md

Installs necessary Python libraries for model handling and acceleration using pip.

```bash
pip install transformers accelerate peft
```

--------------------------------

### Manual Multi-Node Setup

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/evaluation/lm-evaluation-harness/references/distributed-eval.md

Manually configures multi-node evaluation by specifying master node IP and rank. Requires environment variables for coordination.

```bash
accelerate launch \
  --multi_gpu \
  --num_machines 4 \
  --num_processes 32 \
  --main_process_ip $MASTER_IP \
  --main_process_port 29500 \
  --machine_rank $NODE_RANK \
  -m lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu
```

--------------------------------

### Install HuggingFace Tokenizers

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/evaluation/huggingface-tokenizers/SKILL.md

Commands to install the tokenizers library via pip, optionally including transformers for seamless integration.

```bash
pip install tokenizers
pip install tokenizers transformers
```

--------------------------------

### Install YC-Bench Environment

Source: https://github.com/math-inc/opengauss/blob/main/environments/benchmarks/yc_bench/README.md

Instructions for installing the YC-Bench package via pip or from the source repository to enable environment evaluation.

```bash
# Install yc-bench (optional dependency)
pip install "gauss-agent[yc-bench]"

# Or install from source
git clone https://github.com/collinear-ai/yc-bench
cd yc-bench && pip install -e .

# Verify
yc-bench --help
```

--------------------------------

### Faster Package Installs with uv

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/cloud/modal/references/advanced-usage.md

Illustrates using `uv` for faster Python package installations within a Modal image. `uv` is a modern, fast package installer that can significantly speed up build times.

```python
image = modal.Image.debian_slim().uv_pip_install(
    "torch", "transformers", "accelerate"
)
```

--------------------------------

### Install PyTorch Lightning

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/training/pytorch-lightning/SKILL.md

Installs the PyTorch Lightning library using pip. This is the first step to using the framework.

```bash
pip install lightning
```

--------------------------------

### Install or Disable xformers in AudioCraft

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/models/audiocraft/references/troubleshooting.md

Demonstrates how to install the 'xformers' library for memory efficiency or disable its usage within AudioCraft by setting an environment variable.

```bash
# Install xformers for memory efficiency
pip install xformers

# Or disable xformers
export AUDIOCRAFT_USE_XFORMERS=0

# In Python
import os
os.environ["AUDIOCRAFT_USE_XFORMERS"] = "0"
from audiocraft.models import MusicGen
```

--------------------------------

### Install Instructor library via pip

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/instructor/SKILL.md

Commands to install the base Instructor package and specific provider integrations.

```bash
pip install instructor
pip install "instructor[anthropic]"
pip install "instructor[openai]"
pip install "instructor[all]"
```

--------------------------------

### Installing TensorRT-LLM Benchmark Tool

Source: https://github.com/math-inc/opengauss/blob/main/skills/mlops/inference/tensorrt-llm/references/optimization.md

Provides the command to install the TensorRT-LLM benchmark tool. This tool is essential for evaluating the performance of optimized LLM inference.

```bash
# Install benchmark tool
pip install tensorrt_llm[benchmark]
```