PrivateGPT

https://github.com/zylon-ai/private-gpt
Admin
PrivateGPT is a production-ready AI project that enables querying documents using Large Language...
Tokens:21,658
Snippets:289
Trust Score:8.8
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies for Specific Setup

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/concepts.mdx

Use poetry to install dependencies for a specific setup, such as local UI, Qdrant vector store, Ollama LLM, and Ollama embeddings. Replace placeholders with your desired extras.

```bash
poetry install --extras "ui vector-stores-qdrant llms-ollama embeddings-ollama"
```

--------------------------------

### Run LlamaCPP CPU Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Starts the local setup requiring a Hugging Face token.

```sh
HF_TOKEN=<your_hf_token> docker-compose --profile llamacpp-cpu up
```

--------------------------------

### Install Chroma Dependencies

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Install the required extras for Chroma support.

```bash
poetry install --extras chroma
```

--------------------------------

### Install PrivateGPT with Poetry

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Commands to clone the repository and install dependencies with Ollama support.

```bash
# Clone the repository
git clone https://github.com/zylon-ai/private-gpt
cd private-gpt

# Install with Ollama support (recommended local setup)
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"

# Start Ollama service (in a separate terminal)
ollama serve

# Pull required models
ollama pull llama3.1
ollama pull nomic-embed-text

# Run PrivateGPT
PGPT_PROFILES=ollama make run
```

--------------------------------

### Install make on Windows

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install the 'make' utility on Windows using Chocolatey. This is required for running various scripts.

```bash
choco install make
```

--------------------------------

### Install Milvus Dependencies

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Install the required extras for Milvus support.

```bash
poetry install --extras vector-stores-milvus
```

--------------------------------

### Install and Run PrivateGPT with Sagemaker

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install dependencies for Sagemaker support and execute the application using the Sagemaker profile.

```bash
poetry install --extras "ui llms-sagemaker embeddings-sagemaker vector-stores-qdrant"
```

```bash
PGPT_PROFILES=sagemaker make run
```

--------------------------------

### Install and Run PrivateGPT with OpenAI

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install dependencies for OpenAI support and execute the application using the OpenAI profile.

```bash
poetry install --extras "ui llms-openai embeddings-openai vector-stores-qdrant"
```

```bash
PGPT_PROFILES=openai make run
```

--------------------------------

### Local LLM Configuration Example

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Example configuration for the 'llm' section in settings.yaml to run a local model. Adjust 'max_new_tokens' to manage LLM output length.

```yaml
llm:
  mode: local
  max_new_tokens: 256
```

--------------------------------

### Install PrivateGPT with Custom Extras

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install PrivateGPT with specific dependencies using Poetry extras. Replace `<extra1> <extra2>...` with the desired module extras.

```bash
poetry install --extras "<extra1> <extra2>..."
```

--------------------------------

### Verify GPU support

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Example output indicating successful BLAS initialization.

```text
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 |
```

--------------------------------

### Install Xcode command line tools

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Command to install necessary build tools on macOS.

```bash
xcode-select --install
```

--------------------------------

### Install PGVector dependencies

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Install the necessary PostgreSQL vector store extras using poetry.

```bash
poetry install --extras vector-stores-postgres
```

--------------------------------

### Download LLM Models for LlamaCPP

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/concepts.mdx

Execute the setup script to download LLM models required for LlamaCPP to function. This script places the models in the 'models' folder.

```bash
poetry run python scripts/setup
```

--------------------------------

### OpenAI LLM Configuration Example

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Example configuration for 'settings-openai.yaml' to use OpenAI. Set 'mode' to 'openai' and provide API details. The API key can also be set via the OPENAI_API_KEY environment variable.

```yaml
llm:
  mode: openai

openai:
  api_base: <openai-api-base-url> # Defaults to https://api.openai.com/v1
  api_key: <your_openai_api_key>  # You could skip this configuration and use the OPENAI_API_KEY env var instead
  model: <openai_model_to_use> # Optional model to use. Default is "gpt-3.5-turbo"
                               # Note: Open AI Models are listed here: https://platform.openai.com/docs/models
```

--------------------------------

### Install and Run PrivateGPT with Ollama

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install dependencies for Ollama support and execute the application using the Ollama profile.

```bash
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"
```

```bash
PGPT_PROFILES=ollama make run
```

--------------------------------

### Install make on macOS

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install the 'make' utility on macOS using Homebrew. This is required for running various scripts.

```bash
brew install make
```

--------------------------------

### Install llama-cpp-python with CUDA on Linux

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Execute this command in the terminal to install the llama-cpp-python package with CUDA support on Linux. Ensure a C++ compiler and CUDA toolkit are installed.

```bash
CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python numpy==1.26.0
```

--------------------------------

### Install Reranking Dependencies

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/reranker.mdx

Install the necessary dependencies for the cross-encoder reranker using poetry.

```bash
poetry install --extras rerank-sentence-transformers
```

--------------------------------

### Install PrivateGPT with local Llama-CPP extras

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Installs PrivateGPT with extras for local Llama-CPP LLM, HuggingFace embeddings, and Qdrant vector store. Models must be downloaded separately.

```bash
poetry install --extras "ui llms-llama-cpp embeddings-huggingface vector-stores-qdrant"
```

--------------------------------

### Install Postgres Dependencies

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/nodestore.mdx

Install the required extra package for Postgres support using poetry.

```bash
poetry install --extras storage-nodestore-postgres
```

--------------------------------

### Install bitsandbytes for ROCm

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Compiles and installs bitsandbytes from source with ROCm support.

```bash
PYTORCH_ROCM_ARCH=gfx900,gfx906,gfx908,gfx90a,gfx1030,gfx1100,gfx1101,gfx940,gfx941,gfx942
BITSANDBYTES_VERSION=62353b0200b8557026c176e74ac48b84b953a854
git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6
cd bitsandbytes-rocm-5.6
git checkout ${BITSANDBYTES_VERSION}
make hip ROCM_TARGET=${PYTORCH_ROCM_ARCH} ROCM_HOME=/opt/rocm/
pip install . --extra-index-url https://download.pytorch.org/whl/nightly
```

--------------------------------

### Run PrivateGPT with Ollama Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Start PrivateGPT using the Ollama configuration by setting the PGPT_PROFILES environment variable.

```bash
PGPT_PROFILES=ollama make run
```

```bash
PGPT_PROFILES=ollama poetry run python -m private_gpt
```

--------------------------------

### Install llama-cpp-python with GPU support

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Installs llama-cpp-python with HIPBLAS enabled for AMD GPU acceleration.

```bash
LLAMA_CPP_PYTHON_VERSION=0.2.56
DAMDGPU_TARGETS=gfx900;gfx906;gfx908;gfx90a;gfx1030;gfx1100;gfx1101;gfx940;gfx941;gfx942
CMAKE_ARGS="-DLLAMA_HIPBLAS=ON -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DAMDGPU_TARGETS=${DAMDGPU_TARGETS}" poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python==${LLAMA_CPP_PYTHON_VERSION}
```

--------------------------------

### Install ClickHouse Vector Store Extra

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Install the necessary extra for ClickHouse vector store support using poetry.

```bash
poetry install --extras vector-stores-clickhouse
```

--------------------------------

### Install llama-cpp-python with CUDA on Windows

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Use this command in PowerShell to install the llama-cpp-python package with CUDA support on Windows. Ensure all prerequisites like VS2022 and CUDA toolkit are installed.

```powershell
$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python numpy==1.26.0
```

--------------------------------

### Install PrivateGPT with Azure OpenAI extras

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Installs PrivateGPT with specific extras for Azure OpenAI LLM, embeddings, and Qdrant vector store. Ensure Azure OpenAI credentials are configured.

```bash
poetry install --extras "ui llms-azopenai embeddings-azopenai vector-stores-qdrant"
```

--------------------------------

### Install PyTorch for ROCm

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Downloads and installs the specific PyTorch wheel compatible with ROCm 6.0.

```bash
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-6.0/torch-2.1.1%2Brocm6.0-cp311-cp311-linux_x86_64.whl
poetry run pip install --force-reinstall --no-cache-dir torch-2.1.1+rocm6.0-cp311-cp311-linux_x86_64.whl
```

--------------------------------

### Example Usage of PrivateGPT Python SDK

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Demonstrates a typical workflow: ingesting a document, obtaining its IDs, chatting with the document's context, and retrieving relevant chunks based on a query.

```python
# Example usage
if __name__ == "__main__":
    # Ingest a document
    result = ingest_document("quarterly_report.pdf")
    doc_ids = [doc["doc_id"] for doc in result["data"]]
    print(f"Ingested {len(doc_ids)} document chunks")

    # Chat with document context
    messages = [
        {"role": "system", "content": "Answer based on the provided documents."},
        {"role": "user", "content": "What were the key findings in the report?"}
    ]
    response = chat_with_context(messages, use_context=True, doc_ids=doc_ids)
    print(response["choices"][0]["message"]["content"])

    # Get relevant chunks
    chunks = get_relevant_chunks("revenue growth")
    for chunk in chunks:
        print(f"Score: {chunk['score']:.2f} - {chunk['text'][:100]}...")
```

--------------------------------

### Run Ollama Nvidia CUDA Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Starts the service with GPU acceleration enabled via CUDA.

```sh
docker-compose --profile ollama-cuda up
```

--------------------------------

### Run Default Ollama CPU Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Commands to start the Ollama CPU profile using pre-built images.

```sh
docker-compose up
```

```sh
docker-compose --profile ollama-cpu up
```

--------------------------------

### Configure ClickHouse Settings in settings.yaml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Example configuration for ClickHouse in the settings.yaml file, specifying database connection details.

```yaml
vectorstore:
  database: clickhouse

clickhouse:
  host: localhost
  port: 8443
  username: admin
  password: <PASSWORD>
  database: embeddings
  secure: false
```

--------------------------------

### Run PrivateGPT with OpenAI

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Starts PrivateGPT using OpenAI for LLM and embedding models. Requires a 'settings-openai.yaml' profile. This command enables both the API and Gradio UI.

```bash
PGPT_PROFILES=openai make run
```

```bash
PGPT_PROFILES=openai poetry run python -m private_gpt
```

--------------------------------

### Configure PGVector in settings.yaml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Example configuration block for connecting to a PostgreSQL database.

```yaml
vectorstore:
  database: postgres

postgres:
  host: localhost
  port: 5432
  database: postgres
  user: postgres
  password: <PASSWORD>
  schema_name: private_gpt
```

--------------------------------

### Verify llama-cpp-python GPU Installation (Windows)

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

This console output indicates a successful installation of llama-cpp-python with GPU support, showing BLAS enabled and VRAM usage.

```console
llama_new_context_with_model: total VRAM used: 4857.93 MB (model: 4095.05 MB, context: 762.87 MB)
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
```

--------------------------------

### Clone PrivateGPT Repository

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Clone the official PrivateGPT repository and navigate into the project directory. This is the first step to start using PrivateGPT.

```bash
git clone https://github.com/zylon-ai/private-gpt
cd private-gpt
```

--------------------------------

### Install Python 3.11 with pyenv (macOS/Linux)

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Install and set Python version 3.11 using pyenv. Earlier Python versions are not supported.

```bash
pyenv install 3.11
pyenv local 3.11
```

--------------------------------

### Run PrivateGPT with OpenAI-Compatible API

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Starts PrivateGPT using an OpenAI-compatible API server (e.g., vLLM, LocalAI). Requires a profile like 'settings-vllm.yaml' with 'mode: openailike'.

```bash
PGPT_PROFILES=vllm make run
```

--------------------------------

### Manage Ollama Service and Models

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Commands to start the Ollama inference server and manually pull required models.

```bash
ollama serve
```

```bash
ollama pull llama3.1
ollama pull nomic-embed-text
```

--------------------------------

### Summarize Text with Custom Instructions

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Use the /v1/summarize endpoint to get a summary of provided text, with options to specify instructions for the summary's focus and format. Set 'stream' to false for a single response.

```bash
curl -X POST "http://localhost:8001/v1/summarize" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "The quarterly financial report shows revenue of $5.2M, up 23% year-over-year. Operating expenses increased to $3.1M due to new hires. Net income was $1.4M. Customer acquisition cost decreased by 15% while lifetime value increased by 20%.",
    "instructions": "Focus on key financial metrics and growth indicators. Use bullet points.",
    "stream": false
  }'
```

--------------------------------

### Install llama-cpp-python with Metal support for OSX

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Installs or re-installs the llama-cpp-python package with Metal GPU acceleration enabled for macOS. This requires a C++ compiler and Metal libraries.

```bash
CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python
```

--------------------------------

### Run PrivateGPT with Local Models

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Starts PrivateGPT using local LLM and embedding models. Ensure local LLM requirements are met. This command enables both the API and Gradio UI.

```bash
PGPT_PROFILES=local make run
```

```bash
PGPT_PROFILES=local poetry run python -m private_gpt
```

--------------------------------

### Set ARCHFLAGS for Intel Mac

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Workaround for clang architecture errors on Intel-based Macs during package installation.

```bash
ARCHFLAGS="-arch x86_64" pip3 install -r requirements.txt
```

--------------------------------

### Upgrade Poetry to Latest Tested Version

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Upgrade Poetry to version 1.8.3 to avoid known bugs in earlier versions. This command should be run after initial Poetry installation.

```bash
poetry self update 1.8.3
```

--------------------------------

### Clone PrivateGPT Repository

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Initial step to download the project and enter the directory.

```sh
git clone https://github.com/zylon-ai/private-gpt.git
cd private-gpt
```

--------------------------------

### Configure Local Qdrant Path

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Set the persistence path for disk-based Qdrant storage.

```yaml
qdrant:
  path: local_data/private_gpt/qdrant
```

--------------------------------

### Run PrivateGPT with Docker Compose

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Commands to launch the service using different hardware profiles.

```bash
# Run with Ollama CPU mode
docker compose --profile ollama-cpu up

# Run with Ollama CUDA GPU mode
docker compose --profile ollama-cuda up

# Run with LlamaCPP CPU mode
docker compose --profile llamacpp-cpu up
```

--------------------------------

### Run PrivateGPT with Ollama Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/concepts.mdx

Load configuration from default settings and the 'ollama' profile by setting the PGPT_PROFILES environment variable. This allows overriding default settings with Ollama-specific configurations.

```bash
PGPT_PROFILES=ollama make run
```

--------------------------------

### Configure Local LlamaCPP Backend

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

YAML configuration for setting up PrivateGPT with a local LlamaCPP backend.

```yaml
server:
  env_name: ${APP_ENV:local}

llm:
  mode: llamacpp
  max_new_tokens: 512
  context_window: 3900
  tokenizer: meta-llama/Meta-Llama-3.1-8B-Instruct
  prompt_style: "llama3"

llamacpp:
  llm_hf_repo_id: lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF
  llm_hf_model_file: Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

embedding:
  mode: huggingface

huggingface:
  embedding_hf_model_name: nomic-ai/nomic-embed-text-v1.5

vectorstore:
  database: qdrant

qdrant:
  path: local_data/private_gpt/qdrant
```

--------------------------------

### Completion with system prompt and document context

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Execute a completion using a custom system prompt and retrieved document context.

```bash
curl -X POST "http://localhost:8001/v1/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What are the main points of the document?",
    "system_prompt": "You are an expert document analyst. Be concise and factual.",
    "stream": false,
    "use_context": true,
    "include_sources": true
  }'
```

--------------------------------

### Configure Parallel Ingestion

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/ingestion.mdx

Optimize ingestion speed by setting `embedding.ingest_mode` to `parallel` and adjusting `embedding.count_workers`. Start with a lower worker count and increase cautiously to avoid memory issues.

```yaml
embedding:
  ingest_mode: parallel
  count_workers: 4
```

--------------------------------

### Configure Simple NodeStore

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/nodestore.mdx

Set the nodestore database property to simple in settings.yaml for lightweight, disk-persisted storage.

```yaml
nodestore:
  database: simple
```

```yaml
nodestore:
  database: simple
```

--------------------------------

### Run PrivateGPT with Sagemaker Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Launch PrivateGPT with the AWS Sagemaker configuration by specifying the PGPT_PROFILES environment variable.

```bash
PGPT_PROFILES=sagemaker make run
```

```bash
PGPT_PROFILES=sagemaker poetry run python -m private_gpt
```

--------------------------------

### Configure Ollama Backend

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

YAML configuration for setting up PrivateGPT with an Ollama backend.

```yaml
server:
  env_name: ${APP_ENV:ollama}

llm:
  mode: ollama
  max_new_tokens: 512
  context_window: 3900
  temperature: 0.1

embedding:
  mode: ollama

ollama:
  llm_model: llama3.1
  embedding_model: nomic-embed-text
  api_base: http://localhost:11434
  embedding_api_base: http://localhost:11434
  keep_alive: 5m
  top_k: 40
  top_p: 0.9
  repeat_penalty: 1.2
  request_timeout: 120.0

vectorstore:
  database: qdrant

qdrant:
  path: local_data/private_gpt/qdrant
```

--------------------------------

### Configure Postgres NodeStore

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/nodestore.mdx

Define the Postgres connection settings in settings.yaml.

```yaml
nodestore:
  database: postgres

postgres:
  host: localhost
  port: 5432
  database: postgres
  user: postgres
  password: <PASSWORD>
  schema_name: private_gpt
```

--------------------------------

### Run PrivateGPT with Azure OpenAI profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Executes PrivateGPT using the 'azopenai' profile, which utilizes Azure OpenAI LLM and embeddings configured in settings-azopenai.yaml. The UI will be available at http://localhost:8001.

```bash
PGPT_PROFILES=azopenai make run
```

--------------------------------

### Run PrivateGPT with Gemini Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Execute PrivateGPT using the 'gemini' profile. This command loads the settings defined in your `settings-gemini.yaml` file.

```bash
PGPT_PROFILES=gemini make run
```

```bash
PGPT_PROFILES=gemini poetry run python -m private_gpt
```

--------------------------------

### Run PrivateGPT with local profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/installation.mdx

Executes PrivateGPT using the 'local' profile, which uses LlamaCPP LLM and HuggingFace embeddings configured in settings-local.yaml. The UI will be available at http://localhost:8001.

```bash
PGPT_PROFILES=local make run
```

--------------------------------

### Load Multiple Profiles

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/settings.mdx

Set the PGPT_PROFILES environment variable with a comma-separated list of profile names to load multiple configuration profiles. Later profiles override earlier ones.

```text
PGPT_PROFILES=local,cuda
```

--------------------------------

### Build Docker Images Locally

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Commands to compile images from the local codebase.

```sh
docker-compose build
```

```sh
docker-compose up --build
```

```sh
docker-compose --profile <profile_name> up --build
```

--------------------------------

### Configure Vectorstore Database

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/vectordb.mdx

Set the database provider in settings.yaml.

```yaml
vectorstore:
  database: qdrant
```

--------------------------------

### Run PrivateGPT with Azure OpenAI Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Execute PrivateGPT using the Azure OpenAI configuration by setting the PGPT_PROFILES environment variable.

```bash
PGPT_PROFILES=azopenai make run
```

```bash
PGPT_PROFILES=azopenai poetry run python -m private_gpt
```

--------------------------------

### Enable Bulk Local Ingestion

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/ingestion.mdx

Use this command to ingest a folder of documents. The `--watch` flag monitors the folder for changes. Ensure `data.local_ingestion.enabled` is set in your settings.

```bash
make ingest /path/to/folder -- --watch
```

--------------------------------

### Set Hugging Face Access Token in settings.yaml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/troubleshooting.mdx

Add your Hugging Face access token to the `settings.yaml` file to access gated models. Ensure the token is correctly formatted.

```yaml
huggingface:
  access_token: <your-token>
```

--------------------------------

### Run PrivateGPT

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/troubleshooting.mdx

Execute the PrivateGPT application using Poetry. This command is typically used after setting necessary environment variables.

```bash
poetry run python -m privategpt
```

--------------------------------

### Ingest a file

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Upload a file for processing into the vector store.

```bash
curl -X POST "http://localhost:8001/v1/ingest/file" \
  -F "file=@/path/to/document.pdf"
```

--------------------------------

### Run Ollama External API Profile

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/quickstart/quickstart.mdx

Configures PrivateGPT to connect to an Ollama instance running on the host machine.

```sh
OLLAMA_HOST=0.0.0.0 ollama serve
```

```sh
docker-compose --profile ollama-api up
```

--------------------------------

### Streaming Summarization

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Enable streaming for summarization by setting 'stream' to true. This is useful for long texts where you want to receive the summary incrementally.

```bash
curl -X POST "http://localhost:8001/v1/summarize" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Long document text here...",
    "stream": true
  }'
```

--------------------------------

### Verify Postgres Tables

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/nodestore.mdx

Check the created tables in the Postgres database using psql.

```sql
postgres=# \dt private_gpt.*
                  List of relations
   Schema    |      Name       | Type  |    Owner     
-------------+-----------------+-------+--------------
 private_gpt | data_docstore   | table | postgres
 private_gpt | data_indexstore | table | postgres

postgres=# 
```

--------------------------------

### Configure Tokenizer Model in settings.yaml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/troubleshooting.mdx

Specify the desired model for the tokenizer in your `settings.yaml` file. This ensures PrivateGPT uses the correct tokenizer for model interactions.

```yaml
llm:
  tokenizer: meta-llama/Meta-Llama-3.1-8B-Instruct
```

--------------------------------

### POST /v1/completions

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Provides a simplified interface for single-prompt completions, optionally using system prompts and document context.

```APIDOC
## POST /v1/completions

### Description
Generates a completion for a single prompt. Internally converts to the chat format.

### Method
POST

### Endpoint
http://localhost:8001/v1/completions

### Request Body
- **prompt** (string) - Required - The text prompt to complete.
- **system_prompt** (string) - Optional - Instructions for the model behavior.
- **stream** (boolean) - Optional - Whether to stream the response.
- **use_context** (boolean) - Optional - Whether to use document context.
- **include_sources** (boolean) - Optional - Whether to include source metadata.
- **context_filter** (object) - Optional - Filter containing docs_ids.

### Request Example
{
  "prompt": "Write a haiku about programming",
  "stream": false,
  "use_context": false
}
```

--------------------------------

### Mock LLM Configuration for Ingestion

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/ingestion.mdx

To avoid running out of memory during ingestion, set `llm.mode` to `mock`. This configuration allows hardware acceleration for embeddings without loading the full LLM.

```yaml
llm:
  mode: mock
embedding:
  mode: local
```

--------------------------------

### Benchmark Ingestion Speed

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/ingestion.mdx

Use this command to wipe existing data and then benchmark the ingestion speed of a folder using the `scripts/ingest_folder.py` script with mock LLM settings.

```bash
# Wipe your local data, to put yourself in a clean state
# This will delete all your ingested documents
make wipe

time PGPT_PROFILES=mock python ./scripts/ingest_folder.py ~/my-dir/to-ingest/
```

--------------------------------

### Summarize text

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Generates a concise summary of the provided text input.

```bash
curl -X POST "http://localhost:8001/v1/summarize" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Machine learning is a branch of artificial intelligence that focuses on building systems that learn from data. These systems improve their performance over time without being explicitly programmed. Common applications include image recognition, natural language processing, and recommendation systems. Deep learning, a subset of machine learning, uses neural networks with many layers to model complex patterns in data.",
    "stream": false
  }'
```

--------------------------------

### Ingest Folder with Logging to File

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Direct the output logs of the 'ingest_folder.py' script to a specified file using the '--log-file' argument. Combine with '--watch' for continuous monitoring.

```bash
python scripts/ingest_folder.py /path/to/documents --log-file ingestion.log --watch
```

--------------------------------

### Basic completion

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Perform a single-prompt completion without conversation history.

```bash
curl -X POST "http://localhost:8001/v1/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Write a haiku about programming",
    "stream": false,
    "use_context": false
  }'
```

--------------------------------

### Python SDK - Chat Completions

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Python functions for interacting with the chat completions endpoint, supporting context and streaming.

```APIDOC
## POST /v1/chat/completions

### Description
Sends a chat request, optionally using document context and supporting streaming responses.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Request Body
- **messages** (array) - Required - An array of message objects, each with `role` (system/user/assistant) and `content`.
- **use_context** (boolean) - Optional - If true, uses ingested documents for context. Defaults to true.
- **context_filter** (object) - Optional - Filters for context. Contains `docs_ids` (array of strings).
- **include_sources** (boolean) - Optional - If true, includes sources in the response.
- **stream** (boolean) - Optional - If true, enables streaming responses. Defaults to false.

### Request Example
```json
{
  "messages": [
    {"role": "system", "content": "Answer based on the provided documents."},
    {"role": "user", "content": "What were the key findings in the report?"}
  ],
  "use_context": true,
  "context_filter": {"docs_ids": ["doc-id-1"]},
  "include_sources": true,
  "stream": false
}
```

### Response
#### Success Response (200)
- **choices** (array) - An array of completion choices, each containing a `message` object with `content` and `role`.
- **usage** (object) - Information about token usage.

#### Streaming Response (200)
- Server-Sent Events stream of message chunks.
```

--------------------------------

### Configure Reranking in settings.yaml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/reranker.mdx

Update the rag section in settings.yaml to enable reranking and define the retrieval parameters.

```yaml
rag:
  similarity_top_k: 10  # Number of documents to retrieve and consider for reranking
  rerank:
    enabled: true
    top_n: 3  # Number of top-ranked documents to use for generating the answer
```

--------------------------------

### Ingest All Files from a Folder

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Utilize the 'ingest_folder.py' script to process all documents within a specified directory. This is the basic command for bulk ingestion.

```bash
python scripts/ingest_folder.py /path/to/documents
```

--------------------------------

### Check PrivateGPT Service Health

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Use the /health endpoint to verify if the PrivateGPT service is running and responsive. A 'status: ok' response indicates the service is operational.

```bash
# Check service health
curl -X GET "http://localhost:8001/health"
```

```json
# Response:
# {
#   "status": "ok"
# }
```

--------------------------------

### Set Hugging Face Access Token via Environment Variable

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/installation/troubleshooting.mdx

Alternatively, set the `HF_TOKEN` environment variable to provide your Hugging Face access token. This is useful for non-interactive environments.

```bash
export HF_TOKEN=<your-token>
```

--------------------------------

### Log Ingestion to File

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/ingestion.mdx

This command ingests documents from a folder and logs processed and failed files to a specified log file. Use the `--watch` flag to monitor for changes.

```bash
make ingest /path/to/folder -- --watch --log-file /path/to/log/file.log
```

--------------------------------

### Ingest Folder with File Watching

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Enable automatic ingestion of new files added to a folder by using the '--watch' flag with the 'ingest_folder.py' script. This ensures your data is always up-to-date.

```bash
python scripts/ingest_folder.py /path/to/documents --watch
```

--------------------------------

### Gemini Configuration for PrivateGPT

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Create this YAML file to configure PrivateGPT to use Gemini for LLM and embeddings. You can set your API key directly or use the GEMINI_API_KEY environment variable. Optional models for LLM and embeddings can also be specified.

```yaml
llm:
  mode: gemini

embedding:
  mode: gemini

gemini:
  api_key: <your_gemini_api_key>                # You could skip this configuration and use the GEMINI_API_KEY env var instead
  model: <gemini_model_to_use>                  # Optional model to use. Default is models/gemini-pro"
  embedding_model: <gemini_embeddings_to_use>   # Optional model to use. Default is "models/embedding-001"
```

--------------------------------

### Ingest Document File using Python SDK

Source: https://context7.com/zylon-ai/private-gpt/llms.txt

Use the Python SDK's 'ingest_document' function to upload a single file for processing. It handles opening the file in binary mode and sending it via POST request.

```python
import requests
from typing import Generator

BASE_URL = "http://localhost:8001"

def ingest_document(file_path: str) -> dict:
    """Ingest a document file into PrivateGPT."""
    with open(file_path, "rb") as f:
        response = requests.post(
            f"{BASE_URL}/v1/ingest/file",
            files={"file": f}
        )
    response.raise_for_status()
    return response.json()
```

--------------------------------

### Set PGPT_PROFILES on Linux/macOS

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/settings.mdx

Use this command to set the PGPT_PROFILES environment variable on Linux and macOS to select a specific configuration profile.

```bash
export PGPT_PROFILES=my_profile_name_here
```

--------------------------------

### Configure Ollama Settings

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/docs/pages/manual/llms.mdx

Use this YAML configuration for PrivateGPT when integrating with Ollama for LLM and embeddings. Specify the desired Ollama model and optionally the API base URL.

```yaml
llm:
  mode: ollama

ollama:
  model: <ollama_model_to_use> # Required Model to use.
                               # Note: Ollama Models are listed here: https://ollama.ai/library
                               #       Be sure to pull the model to your Ollama server
  api_base: <ollama-api-base-url> # Defaults to http://localhost:11434
```

--------------------------------

### Add New Page Configuration in docs.yml

Source: https://github.com/zylon-ai/private-gpt/blob/main/fern/README.md

This YAML snippet demonstrates how to add a new page to the documentation by defining its display name and relative path within the `docs.yml` configuration file. Ensure the path correctly points to your Markdown or MDX file.

```yaml
navigation:
  # ...
  - tab: my-existing-tab
    layout:
      # ...
      - section: My Existing Section
        contents:
          # ...
          - page: My new page display name
            # The path of the page, relative to `fern/`
            path: ./docs/pages/my-existing-tab/new-page-content.mdx
```
Tokens:
Raw