GPT4All

https://docs.gpt4all.io
Admin
GPT4All runs large language models privately on everyday desktops and laptops without requiring API...
Tokens:26,706
Snippets:185
Trust Score:9.9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install GPT4All Python SDK

Source: https://docs.gpt4all.io/index.html

Use pip to install the necessary package for the Python SDK.

```bash
pip install gpt4all
```

--------------------------------

### Install OpenLIT

Source: https://docs.gpt4all.io/gpt4all_python/monitoring.html

Install the OpenLIT library using pip. This is a prerequisite for enabling monitoring.

```bash
pip install openlit

```

--------------------------------

### React App Setup Commands

Source: https://docs.gpt4all.io/gpt4all_desktop/chats.html

Terminal commands to initialize a new React project.

```bash
npx create-react-app guessing-game
cd guessing-game

```

--------------------------------

### Install GPT4All Package

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Install the gpt4all package using pip. It is recommended to do this within a virtual environment.

```bash
pip install gpt4all

```

--------------------------------

### Legacy System Prompt Example

Source: https://docs.gpt4all.io/gpt4all_desktop/chat_templates.html

An example of a system prompt containing control tokens that are no longer supported in GPT4All v3.5+.

```text
<|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>

```

--------------------------------

### Initialize OpenLIT and Run GPT4All

Source: https://docs.gpt4all.io/gpt4all_python/monitoring.html

Initialize OpenLIT for monitoring and then use the GPT4All library to run LLM queries. The `openlit.init()` call starts the monitoring process. Optionally, `collect_gpu_stats=True` can be passed to `init` to gather GPU metrics.

```python
from gpt4all import GPT4All
import openlit

openlit.init()  # start
# openlit.init(collect_gpu_stats=True)  # Optional: To configure GPU monitoring

model = GPT4All(model_name='orca-mini-3b-gguf2-q4_0.gguf')

# Start a chat session and send queries
with model.chat_session():
    response1 = model.generate(prompt='hello', temp=0)
    response2 = model.generate(prompt='write me a short poem', temp=0)
    response3 = model.generate(prompt='thank you', temp=0)

    print(model.current_chat_session)

```

--------------------------------

### LLM Chat Session Generation Example

Source: https://docs.gpt4all.io/gpt4all_python/home.html

This snippet shows how to load a specific LLM and initiate a chat session to generate a response to a prompt, applying chat templates for helpfulness.

```python
from gpt4all import GPT4All
model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf")
with model.chat_session():
    print(model.generate("quadratic formula"))

```

--------------------------------

### Generate Chat Completions with PowerShell

Source: https://docs.gpt4all.io/gpt4all_api_server/home.html

Example of how to generate chat completions using the GPT4All API server with PowerShell. Ensure the API server is enabled and running.

```powershell
Invoke-WebRequest -URI http://localhost:4891/v1/chat/completions -Method POST -ContentType application/json -Body '{ 
"model": "Phi-3 Mini Instruct",
"messages": [{"role":"user","content":"Who is Lionel Messi?"}],
"max_tokens": 50,
"temperature": 0.28
}'
```

--------------------------------

### GET /list_gpus

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Retrieves a list of available GPU devices.

```APIDOC
## GET /list_gpus

### Description
List the names of the available GPU devices.

### Method
GET

### Endpoint
/list_gpus

### Response
#### Success Response (200)
- **devices** (list[str]) - A list of strings representing the names of the available GPU devices.
```

--------------------------------

### Direct LLM Generation Example

Source: https://docs.gpt4all.io/gpt4all_python/home.html

This snippet demonstrates direct generation using `model.generate()` without a chat session, which may result in responses that are less like helpful assistants and more like data continuations.

```python
from gpt4all import GPT4All
model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf")
print(model.generate("quadratic formula"))

```

--------------------------------

### Embed4All SBert Model Usage

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Example of how to instantiate the `Embed4All` class with the SBert model for local embedding generation.

```python
Embed4All("all-MiniLM-L6-v2.gguf2.f16.gguf")
```

--------------------------------

### Prompting Llama 3 for Explanations

Source: https://docs.gpt4all.io/gpt4all_desktop/chats.html

Example prompt for Llama 3 to explain scientific concepts to a child.

```text
explain why the sky is blue in a way that is correct and makes sense to a child

```

--------------------------------

### Load Tokenizer from Local Path (Gated Models)

Source: https://docs.gpt4all.io/gpt4all_desktop/chat_templates.html

For gated models, after cloning the repository and selectively pulling necessary files, you can load the tokenizer from the local directory. This requires git and git-lfs to be installed.

```bash
$ GIT_LFS_SKIP_SMUDGE=1 git clone hf.co:meta-llama/Llama-3.1-8B-Instruct.git
$ cd Llama-3.1-8B-Instruct
$ git lfs pull -I "tokenizer.*"
```

```python
tokenizer = AutoTokenizer.from_pretrained('.')
```

--------------------------------

### Generate Chat Completions with cURL

Source: https://docs.gpt4all.io/gpt4all_api_server/home.html

Example of how to generate chat completions using the GPT4All API server with cURL. Ensure the API server is enabled and running.

```bash
curl -X POST http://localhost:4891/v1/chat/completions -d '{
"model": "Phi-3 Mini Instruct",
"messages": [{"role":"user","content":"Who is Lionel Messi?"}],
"max_tokens": 50,
"temperature": 0.28
}'
```

--------------------------------

### Load and Use LLM for Chat

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Load an LLM by its name using the GPT4All class. The model will be downloaded if it's the first time loading. This example demonstrates initiating a chat session and generating a response.

```python
from gpt4all import GPT4All
model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf") # downloads / loads a 4.66GB LLM
with model.chat_session():
    print(model.generate("How can I run LLMs efficiently on my laptop?", max_tokens=1024))

```

--------------------------------

### Prompting Nous Hermes 2 for React App

Source: https://docs.gpt4all.io/gpt4all_desktop/chats.html

Example prompt for generating a simple React guessing game.

```text
write me a react app i can run from the command line to play a quick game

```

--------------------------------

### Get LLModel Backend

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Returns the name of the llama.cpp backend currently in use. Possible values are 'cpu', 'kompute', 'cuda', or 'metal'.

```python
@property
def backend(self) -> Literal["cpu", "kompute", "cuda", "metal"]:
    """The name of the llama.cpp backend currently in use. One of "cpu", "kompute", "cuda", or "metal"."""
    return self.model.backend
```

--------------------------------

### Extract Chat Template using CLI

Source: https://docs.gpt4all.io/gpt4all_desktop/chat_templates.html

Use this command-line tool to extract the chat template from a tokenizer_config.json file. Ensure jq is installed and the file is in the current directory.

```bash
jq -r ".chat_template" tokenizer_config.json
```

```bash
jq -r ".chat_template" tokenizer_config.json >chat_template.txt
```

--------------------------------

### Embed4All Nomic Embed v1.5 Model Usage

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Example of how to instantiate the `Embed4All` class with the Nomic Embed v1.5 model for local embedding generation.

```python
Embed4All("nomic-embed-text-v1.5.f16.gguf")
```

--------------------------------

### Embed4All Nomic Embed v1 Model Usage

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Example of how to instantiate the `Embed4All` class with the Nomic Embed v1 model for local embedding generation.

```python
Embed4All("nomic-embed-text-v1.f16.gguf")
```

--------------------------------

### Get Chat Template using Python (Open Models)

Source: https://docs.gpt4all.io/gpt4all_desktop/chat_templates.html

This Python code snippet retrieves the chat template for open HuggingFace models using the transformers library. Ensure transformers is installed and updated.

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('NousResearch/Hermes-2-Pro-Llama-3-8B')
print(tokenizer.get_chat_template())
```

```python
open('chat_template.txt', 'w').write(tokenizer.get_chat_template())
```

--------------------------------

### Initialize GPT4All Class

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

The constructor handles model loading, device selection, and configuration of execution parameters like context size and thread count.

```python
class GPT4All:
    """
    Python class that handles instantiation, downloading, generation and chat with GPT4All models.
    """

    def __init__(
        self,
        model_name: str,
        *,
        model_path: str | os.PathLike[str] | None = None,
        model_type: str | None = None,
        allow_download: bool = True,
        n_threads: int | None = None,
        device: str | None = None,
        n_ctx: int = 2048,
        ngl: int = 100,
        verbose: bool = False,
    ):
        """
        Constructor

        Args:
            model_name: Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
            model_path: Path to directory containing model file or, if file does not exist, where to download model.
                Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
            model_type: Model architecture. This argument currently does not have any functionality and is just used as
                descriptive identifier for user. Default is None.
            allow_download: Allow API to download models from gpt4all.io. Default is True.
            n_threads: number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
            device: The processing unit on which the GPT4All model will run. It can be set to:
                - "cpu": Model will run on the central processing unit.
                - "gpu": Use Metal on ARM64 macOS, otherwise the same as "kompute".
                - "kompute": Use the best GPU provided by the Kompute backend.
                - "cuda": Use the best GPU provided by the CUDA backend.
                - "amd", "nvidia": Use the best GPU provided by the Kompute backend from this vendor.
                - A specific device name from the list returned by `GPT4All.list_gpus()`.
                Default is Metal on ARM64 macOS, "cpu" otherwise.

                Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
            n_ctx: Maximum size of context window
            ngl: Number of GPU layers to use (Vulkan)
            verbose: If True, print debug messages.
        """

        self.model_type = model_type
        self._chat_session: ChatSession | None = None

        device_init = None
        if sys.platform == "darwin":
            if device is None:
                backend = "auto"  # "auto" is effectively "metal" due to currently non-functional fallback
            elif device == "cpu":
                backend = "cpu"
            else:
                if platform.machine() != "arm64" or device != "gpu":
                    raise ValueError(f"Unknown device for this platform: {device}")
                backend = "metal"
        else:
            backend = "kompute"
```

--------------------------------

### Initialize Model Backend

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Internal logic for selecting and initializing the appropriate compute backend based on the provided device.

```python
            raise ValueError(f"Unknown device for this platform: {device}")
            backend = "metal"
    else:
        backend = "kompute"
        if device is None or device == "cpu":
            pass  # use kompute with no device
        elif device in ("cuda", "kompute"):
            backend = device
            device_init = "gpu"
        elif device.startswith("cuda:"):
            backend = "cuda"
            device_init = _remove_prefix(device, "cuda:")
        else:
            device_init = _remove_prefix(device, "kompute:")

    # Retrieve model and download if allowed
    self.config: ConfigType = self.retrieve_model(model_name, model_path=model_path, allow_download=allow_download, verbose=verbose)
    self.model = LLModel(self.config["path"], n_ctx, ngl, backend)
    if device_init is not None:
        self.model.init_gpu(device_init)
    self.model.load_model()
    # Set n_threads
    if n_threads is not None:
        self.model.set_thread_count(n_threads)
```

--------------------------------

### Markdown Representation of Excel Data

Source: https://docs.gpt4all.io/gpt4all_desktop/cookbook/use-local-ai-models-to-privately-chat-with-microsoft-excel.html

Example of how GPT4All formats an Excel spreadsheet into Markdown for LLM consumption.

```markdown
## disney_income_stmt

|Walt Disney Co.|||||||
|---|---|---|---|---|---|---|
|Consolidated Income Statement|||||||
|||||||||
|US$ in millions|||||||
|12 months ended:|2023-09-30 00:00:00|2022-10-01 00:00:00|2021-10-02 00:00:00|2020-10-03 00:00:00|2019-09-28 00:00:00|2018-09-29 00:00:00|
|Services|79562|74200|61768|59265|60542|50869|
...
...
...
```

--------------------------------

### POST /init

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Initializes the GPT4All embedding model instance with specific hardware and threading configurations.

```APIDOC
## POST /init

### Description
Constructor to initialize the embedding model. Sets up CPU threads and device processing units.

### Method
POST

### Parameters
#### Request Body
- **model_name** (str) - Optional - Name of the model to load.
- **n_threads** (int) - Optional - Number of CPU threads to use.
- **device** (str) - Optional - Processing unit (e.g., 'cpu', 'gpu').
```

--------------------------------

### Initialize LLModel with GPU Support

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Initializes an LLModel instance, optionally configuring GPU acceleration. Use this when you need to specify the backend and device for GPU processing.

```python
if device is None or device == "cpu":
    pass  # use kompute with no device
elif device in ("cuda", "kompute"):
    backend = device
    device_init = "gpu"
elif device.startswith("cuda:"):
    backend = "cuda"
    device_init = _remove_prefix(device, "cuda:")
else:
    device_init = _remove_prefix(device, "kompute:")

# Retrieve model and download if allowed
self.config: ConfigType = self.retrieve_model(model_name, model_path=model_path, allow_download=allow_download, verbose=verbose)
self.model = LLModel(self.config["path"], n_ctx, ngl, backend)
if device_init is not None:
    self.model.init_gpu(device_init)
self.model.load_model()
# Set n_threads
if n_threads is not None:
    self.model.set_thread_count(n_threads)
```

--------------------------------

### Nomic Embed v1 Usage

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Example of how to use the Nomic Embed v1 model with the `nomic` library for local inference.

```python
embed.text(strings, model="nomic-embed-text-v1", inference_mode="local")
```

--------------------------------

### GPT4All Class Constructor

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

The constructor initializes the GPT4All model instance with specific hardware and context settings. Ensure the selected device has sufficient RAM to avoid initialization errors.

```python
def __init__(
    self,
    model_name: str,
    *,
    model_path: str | os.PathLike[str] | None = None,
    model_type: str | None = None,
    allow_download: bool = True,
    n_threads: int | None = None,
    device: str | None = None,
    n_ctx: int = 2048,
    ngl: int = 100,
    verbose: bool = False,
):
    """
    Constructor

    Args:
        model_name: Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
        model_path: Path to directory containing model file or, if file does not exist, where to download model.
            Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
        model_type: Model architecture. This argument currently does not have any functionality and is just used as
            descriptive identifier for user. Default is None.
        allow_download: Allow API to download models from gpt4all.io. Default is True.
        n_threads: number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
        device: The processing unit on which the GPT4All model will run. It can be set to:
            - "cpu": Model will run on the central processing unit.
            - "gpu": Use Metal on ARM64 macOS, otherwise the same as "kompute".
            - "kompute": Use the best GPU provided by the Kompute backend.
            - "cuda": Use the best GPU provided by the CUDA backend.
            - "amd", "nvidia": Use the best GPU provided by the Kompute backend from this vendor.
            - A specific device name from the list returned by `GPT4All.list_gpus()`.
            Default is Metal on ARM64 macOS, "cpu" otherwise.

            Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
        n_ctx: Maximum size of context window
        ngl: Number of GPU layers to use (Vulkan)
        verbose: If True, print debug messages.
    """

    self.model_type = model_type
    self._chat_session: ChatSession | None = None

    device_init = None
    if sys.platform == "darwin":
        if device is None:
            backend = "auto"  # "auto" is effectively "metal" due to currently non-functional fallback
        elif device == "cpu":
            backend = "cpu"
        else:
            if platform.machine() != "arm64" or device != "gpu":

```

--------------------------------

### Nomic Embed v1.5 Usage

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Example of how to use the Nomic Embed v1.5 model with the `nomic` library for local inference.

```python
embed.text(strings, model="nomic-embed-text-v1.5", inference_mode="local")
```

--------------------------------

### Get Current Chat Session History

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Retrieves the history of the current chat session. Returns None if no chat session is active.

```python
@property
def current_chat_session(self) -> list[MessageType] | None:
    return None if self._chat_session is None else self._chat_session.history
```

--------------------------------

### LLModel Configuration and Properties

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Details on configuring the LLModel, including device selection, backend, and accessing model properties.

```APIDOC
## LLModel Initialization

### Description
Initializes the LLModel with a given configuration, context size, GPU layers, and backend settings.

### Method
Constructor

### Endpoint
(Class method)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **config_path** (string) - Required - Path to the model configuration.
- **n_ctx** (integer) - Required - The context size for the model.
- **ngl** (integer) - Optional - Number of GPU layers to use. Defaults to 0 (CPU).
- **backend** (string) - Optional - The backend to use (e.g., "cpu", "kompute", "cuda", "metal"). Defaults to "cpu".

### Request Example
```python
# Assuming 'config' is a dictionary obtained from retrieve_model
model = LLModel(config['path'], n_ctx=2048, ngl=32, backend='cuda')
```

### Response
None (initializes the model object)

```

```APIDOC
## LLModel Properties

### Description
Access properties of the initialized LLModel, such as the backend, device, and current chat session history.

### Method
Property Access

### Endpoint
(Instance properties)

### Parameters
None

### Properties
- **backend** (string) - Read-only - The name of the llama.cpp backend currently in use (e.g., "cpu", "kompute", "cuda", "metal").
- **device** (string | None) - Read-only - The name of the GPU device currently in use, or `None` for backends other than Kompute or CUDA.
- **current_chat_session** (list[MessageType] | None) - Read/Write - The history of the current chat session. Can be set to update the session history.

### Request Example
```python
print(f"Backend: {model.backend}")
print(f"Device: {model.device}")

# Accessing chat history
if model.current_chat_session:
    print(f"Current chat has {len(model.current_chat_session)} messages.")

# Setting chat history (example)
# model.current_chat_session = [{"role": "user", "content": "Hello"}]
```

### Response
- **backend**: The string name of the backend.
- **device**: The string name of the device or None.
- **current_chat_session**: A list of message objects or None.

```

```APIDOC
## LLModel Methods

### Description
Provides methods for interacting with the loaded LLM, such as initializing the GPU, loading the model, setting thread counts, and closing the model instance.

### Method
Instance Methods

### Endpoint
(Instance methods)

### Parameters
None

### Methods
- **init_gpu(device_id: str)**: Initializes the GPU with the specified device ID.
- **load_model()**: Loads the model into memory.
- **set_thread_count(n_threads: int)**: Sets the number of threads to use for model processing.
- **close()**: Deletes the model instance and frees associated system resources.

### Request Example
```python
# Assuming 'model' is an initialized LLModel object
model.init_gpu("0")  # Initialize GPU device 0
model.load_model()
model.set_thread_count(8)

# ... perform inference ...

model.close() # Clean up resources
```

### Response
None (methods perform actions or modify state)

```

--------------------------------

### Initialize Embed4All

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Instantiate the Embed4All class. You can specify a model name, number of threads, and device. If no model name is provided, a default model is used.

```python
from gpt4all import Embed4All

embed4all = Embed4All()
# or with a specific model
# embed4all = Embed4All(model_name="all-MiniLM-L6-v2.gguf2.f16.gguf")
```

--------------------------------

### Get LLModel GPU Device

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Returns the name of the GPU device currently in use. This is applicable for Kompute or CUDA backends; otherwise, it returns None.

```python
@property
def device(self) -> str | None:
    """The name of the GPU device currently in use, or None for backends other than Kompute or CUDA."""
    return self.model.device
```

--------------------------------

### POST /generate

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Generates text completions from a GPT4All model based on a provided prompt and configuration parameters.

```APIDOC
## POST /generate

### Description
Generates outputs from any GPT4All model based on the input prompt. Supports streaming and various sampling parameters.

### Method
POST

### Parameters
#### Request Body
- **prompt** (str) - Required - The prompt for the model to complete.
- **max_tokens** (int) - Optional (default: 200) - The maximum number of tokens to generate.
- **temp** (float) - Optional (default: 0.7) - The model temperature.
- **top_k** (int) - Optional (default: 40) - Randomly sample from the top_k most likely tokens.
- **top_p** (float) - Optional (default: 0.4) - Randomly sample from the top most likely tokens whose probabilities add up to top_p.
- **min_p** (float) - Optional (default: 0.0) - Randomly sample from the top most likely tokens whose probabilities are at least min_p.
- **repeat_penalty** (float) - Optional (default: 1.18) - Penalize the model for repetition.
- **repeat_last_n** (int) - Optional (default: 64) - How far in the generation history to apply the repeat penalty.
- **n_batch** (int) - Optional (default: 8) - Number of prompt tokens processed in parallel.
- **n_predict** (int) - Optional (default: None) - Equivalent to max_tokens.
- **streaming** (bool) - Optional (default: False) - If True, returns a generator yielding tokens.
- **callback** (function) - Optional - A function to receive tokens as they are generated.

### Request Example
{
  "prompt": "What is the capital of France?",
  "max_tokens": 50,
  "temp": 0.7
}

### Response
#### Success Response (200)
- **result** (Any) - Either the entire completion string or a generator yielding tokens.
```

--------------------------------

### GPT4All Constructor

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Initializes the GPT4All model with specified parameters. This is the primary method for loading and configuring a model.

```APIDOC
## GPT4All Constructor

### Description
Initializes the GPT4All model with specified parameters. This is the primary method for loading and configuring a model.

### Method
`__init__`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **`model_name`** (str) - Required - Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
- **`model_path`** (str | PathLike[str] | None) - Optional, default: `None` - Path to directory containing model file or, if file does not exist, where to download model. Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
- **`model_type`** (str | None) - Optional, default: `None` - Model architecture. This argument currently does not have any functionality and is just used as descriptive identifier for user.
- **`allow_download`** (bool) - Optional, default: `True` - Allow API to download models from gpt4all.io.
- **`n_threads`** (int | None) - Optional, default: `None` - Number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
- **`device`** (str | None) - Optional, default: `None` - The processing unit on which the GPT4All model will run. It can be set to:
    - "cpu": Model will run on the central processing unit.
    - "gpu": Use Metal on ARM64 macOS, otherwise the same as "kompute".
    - "kompute": Use the best GPU provided by the Kompute backend.
    - "cuda": Use the best GPU provided by the CUDA backend.
    - "amd", "nvidia": Use the best GPU provided by the Kompute backend from this vendor.
    - A specific device name from the list returned by `GPT4All.list_gpus()`.
    Default is Metal on ARM64 macOS, "cpu" otherwise.
    Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
- **`n_ctx`** (int) - Optional, default: `2048` - Maximum size of context window.
- **`ngl`** (int) - Optional, default: `100` - Number of GPU layers to use (Vulkan).
- **`verbose`** (bool) - Optional, default: `False` - If True, print debug messages.

### Request Example
```python
from gpt4all import GPT4All

# Example with minimal required parameters
gpt4all_instance = GPT4All(model_name="ggml-model-q4_0.bin")

# Example with advanced configuration
gpt4all_instance_advanced = GPT4All(
    model_name="ggml-model-q4_0.bin",
    model_path="/path/to/models",
    allow_download=False,
    n_threads=8,
    device="gpu",
    n_ctx=4096,
    ngl=64,
    verbose=True
)
```

### Response
#### Success Response (200)
This method does not return a value directly but initializes the GPT4All object.

#### Response Example
None (initialization)

```

--------------------------------

### Initialize GPT4All Model

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Constructor for the GPT4All class. Allows specifying the model name, number of CPU threads, and the processing device. Remaining keyword arguments are passed to the underlying GPT4All constructor.

```python
def __init__(self, model_name: str | None = None, *, n_threads: int | None = None, device: str | None = None, **kwargs: Any):
    """
    Constructor

    Args:
        n_threads: number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
        device: The processing unit on which the embedding model will run. See the `GPT4All` constructor for more info.
        kwargs: Remaining keyword arguments are passed to the `GPT4All` constructor.
    """
    if model_name is None:
        model_name = "all-MiniLM-L6-v2.gguf2.f16.gguf"
    self.gpt4all = GPT4All(model_name, n_threads=n_threads, device=device, **kwargs)
```

--------------------------------

### Generate Embeddings Locally with Nomic

Source: https://docs.gpt4all.io/gpt4all_python/home.html

Use the `nomic` Python library to generate text embeddings locally. This example downloads a model if not present and prints the number of embeddings and their dimensions.

```python
from nomic import embed
embeddings = embed.text(["String 1", "String 2"], inference_mode="local")['embeddings']
print("Number of embeddings created:", len(embeddings))
print("Number of dimensions per embedding:", len(embeddings[0]))

```

--------------------------------

### Run LLM Chat Session

Source: https://docs.gpt4all.io/index.html

Load a model and initiate a chat session to generate responses.

```python
from gpt4all import GPT4All
model = GPT4All("Meta-Llama-3-8B-Instruct.Q4_0.gguf") # downloads / loads a 4.66GB LLM
with model.chat_session():
    print(model.generate("How can I run LLMs efficiently on my laptop?", max_tokens=1024))
```

--------------------------------

### POST /generate

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Generates text completions from a GPT4All model. Supports various generation parameters and streaming output.

```APIDOC
## POST /generate

### Description
Generates text completions from any GPT4All model. This method allows for fine-grained control over the generation process through various parameters and supports streaming responses.

### Method
POST

### Endpoint
/generate

### Parameters
#### Query Parameters
- **prompt** (str) - Required - The prompt for the model to complete.
- **max_tokens** (int) - Optional - The maximum number of tokens to generate. Defaults to 200.
- **temp** (float) - Optional - The model temperature. Larger values increase creativity but decrease factuality. Defaults to 0.7.
- **top_k** (int) - Optional - Randomly sample from the top_k most likely tokens at each generation step. Set this to 1 for greedy decoding. Defaults to 40.
- **top_p** (float) - Optional - Randomly sample at each generation step from the top most likely tokens whose probabilities add up to top_p. Defaults to 0.4.
- **min_p** (float) - Optional - Randomly sample at each generation step from the top most likely tokens whose probabilities are at least min_p. Defaults to 0.0.
- **repeat_penalty** (float) - Optional - Penalize the model for repetition. Higher values result in less repetition. Defaults to 1.18.
- **repeat_last_n** (int) - Optional - How far in the models generation history to apply the repeat penalty. Defaults to 64.
- **n_batch** (int) - Optional - Number of prompt tokens processed in parallel. Larger values decrease latency but increase resource requirements. Defaults to 8.
- **n_predict** (int | None) - Optional - Equivalent to max_tokens, exists for backwards compatibility.
- **streaming** (bool) - Optional - If True, this method will instead return a generator that yields tokens as the model generates them. Defaults to False.
- **callback** (ResponseCallbackType) - Optional - A function with arguments token_id:int and response:str, which receives the tokens from the model as they are generated and stops the generation by returning False. Defaults to an empty_response_callback.

### Request Example
```json
{
  "prompt": "What is the capital of France?",
  "max_tokens": 50,
  "streaming": true
}
```

### Response
#### Success Response (200)
- **completion** (str | Iterable[str]) - The generated text completion, either as a single string or a generator yielding tokens.

#### Response Example
```json
{
  "completion": "The capital of France is Paris."
}
```

#### Streaming Response Example
```json
// Generator yielding tokens
// Example: 'The', ' capital', ' of', ' France', ' is', ' Paris', '.'
```
```

--------------------------------

### Download Model from GPT4All

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Downloads a model file from gpt4all.io, supporting resuming interrupted downloads and verifying integrity with expected size and MD5 hash. Requires `requests`, `tqdm`, and `hashlib`.

```python
import requests
import sys
import os
import hashlib
from pathlib import Path
from urllib3.exceptions import ProtocolError, IncompleteRead
from requests.exceptions import ChunkedEncodingError

# Assume _fsync is defined elsewhere, e.g., os.fsync
def _fsync(f):
    try:
        os.fsync(f.fileno())
    except AttributeError:
        pass

def download_model(model_filename: str,
        model_path: str | os.PathLike[str],
        verbose: bool = True,
        url: str | None = None,
        expected_size: int | None = None,
        expected_md5: str | None = None,
    ) -> str | os.PathLike[str]:
        """
        Download model from gpt4all.io.

        Args:
            model_filename: Filename of model (with .gguf extension).
            model_path: Path to download model to.
            verbose: If True (default), print debug messages.
            url: the models remote url (e.g. may be hosted on HF)
            expected_size: The expected size of the download.
            expected_md5: The expected MD5 hash of the download.

        Returns:
            Model file destination.
        """

        # Download model
        if url is None:
            url = f"https://gpt4all.io/models/gguf/{model_filename}"

        def make_request(offset=None):
            headers = {}
            if offset:
                print(f"\nDownload interrupted, resuming from byte position {offset}", file=sys.stderr)
                headers["Range"] = f"bytes={offset}-"  # resume incomplete response
                headers["Accept-Encoding"] = "identity"  # Content-Encoding changes meaning of ranges
            response = requests.get(url, stream=True, headers=headers)
            if response.status_code not in (200, 206):
                raise ValueError(f"Request failed: HTTP {response.status_code} {response.reason}")
            if offset and (response.status_code != 206 or str(offset) not in response.headers.get("Content-Range", "")):
                raise ValueError("Connection was interrupted and server does not support range requests")
            if (enc := response.headers.get("Content-Encoding")) is not None:
                raise ValueError(f"Expected identity Content-Encoding, got {enc}")
            return response

        response = make_request()

        total_size_in_bytes = int(response.headers.get("content-length", 0))
        block_size = 2**20  # 1 MB

        partial_path = Path(model_path) / (model_filename + ".part")

        with open(partial_path, "w+b") as partf:
            try:
                with tqdm(desc="Downloading", total=total_size_in_bytes, unit="iB", unit_scale=True) as progress_bar:
                    while True:
                        last_progress = progress_bar.n
                        try:
                            for data in response.iter_content(block_size):
                                partf.write(data)
                                progress_bar.update(len(data))
                        except ChunkedEncodingError as cee:
                            if cee.args and isinstance(pe := cee.args[0], ProtocolError):
                                if len(pe.args) >= 2 and isinstance(ir := pe.args[1], IncompleteRead):
                                    assert progress_bar.n <= ir.partial  # urllib3 may be ahead of us but never behind
                                    # the socket was closed during a read - retry
                                    response = make_request(progress_bar.n)
                                    continue
                            raise
                        if total_size_in_bytes != 0 and progress_bar.n < total_size_in_bytes:
                            if progress_bar.n == last_progress:
                                raise RuntimeError("Download not making progress, aborting.")
                            # server closed connection prematurely - retry
                            response = make_request(progress_bar.n)
                            continue
                        break

                # verify file integrity
                file_size = partf.tell()
                if expected_size is not None and file_size != expected_size:
                    raise ValueError(f"Expected file size of {expected_size} bytes, got {file_size}")
                if expected_md5 is not None:
                    partf.seek(0)
                    hsh = hashlib.md5()
                    with tqdm(desc="Verifying", total=file_size, unit="iB", unit_scale=True) as bar:
                        while chunk := partf.read(block_size):
                            hsh.update(chunk)
                            bar.update(len(chunk))
                    if hsh.hexdigest() != expected_md5.lower():
                        raise ValueError(f"Expected MD5 hash of {expected_md5!r}, got {hsh.hexdigest()!r}")
            except:
                if verbose:
                    print("Cleaning up the interrupted download...", file=sys.stderr)
                try:
                    os.remove(partial_path)
                except OSError:
                    pass
                raise

            # flush buffers and sync the inode
            partf.flush()
            _fsync(partf)

        # move to final destination
        download_path = Path(model_path) / model_filename
        try:
            os.rename(partial_path, download_path)
        except FileExistsError:
            # If the file already exists, we assume it's the correct one and skip the download
            # This is a simplification; a more robust solution might compare hashes or sizes.
            if verbose:
                print(f"Model file {download_path} already exists. Skipping download.", file=sys.stderr)
            # Ensure the partial file is removed if the target exists
            try:
                os.remove(partial_path)
            except OSError:
                pass
            return download_path

        return download_path
```

--------------------------------

### GPT4All Class Constructor

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Details the parameters for initializing the GPT4All Python class, which is used for handling GPT4All models.

```APIDOC
## GPT4All Class Constructor

### Description
Initializes the GPT4All Python class, enabling interaction with GPT4All models. This includes setting up model paths, download permissions, and hardware acceleration options.

### Method
__init__

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **model_name** (str) - Required - Name of GPT4All or custom model. Including ".gguf" file extension is optional but encouraged.
- **model_path** (str | os.PathLike[str] | None) - Optional - Path to directory containing model file or, if file does not exist, where to download model. Default is None, in which case models will be stored in `~/.cache/gpt4all/`.
- **model_type** (str | None) - Optional - Model architecture. This argument currently does not have any functionality and is just used as descriptive identifier for user. Default is None.
- **allow_download** (bool) - Optional - Allow API to download models from gpt4all.io. Default is True.
- **n_threads** (int | None) - Optional - number of CPU threads used by GPT4All. Default is None, then the number of threads are determined automatically.
- **device** (str | None) - Optional - The processing unit on which the GPT4All model will run. It can be set to:
    - "cpu": Model will run on the central processing unit.
    - "gpu": Use Metal on ARM64 macOS, otherwise the same as "kompute".
    - "kompute": Use the best GPU provided by the Kompute backend.
    - "cuda": Use the best GPU provided by the CUDA backend.
    - "amd", "nvidia": Use the best GPU provided by the Kompute backend from this vendor.
    - A specific device name from the list returned by `GPT4All.list_gpus()`.
    Default is Metal on ARM64 macOS, "cpu" otherwise.
    Note: If a selected GPU device does not have sufficient RAM to accommodate the model, an error will be thrown, and the GPT4All instance will be rendered invalid. It's advised to ensure the device has enough memory before initiating the model.
- **n_ctx** (int) - Optional - Maximum size of context window. Default is 2048.
- **ngl** (int) - Optional - Number of GPU layers to use (Vulkan). Default is 100.
- **verbose** (bool) - Optional - If True, print debug messages. Default is False.

### Request Example
```python
from gpt4all import GPT4All

gpt4all = GPT4All(
    model_name="ggml-model-q4_0.bin",
    model_path="/path/to/models/",
    n_threads=4,
    device="gpu"
)
```

### Response
#### Success Response (200)
Initializes the GPT4All object.

#### Response Example
```python
# No direct response body for constructor, object is created in memory.
```
```

--------------------------------

### List Available Models

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Fetches the list of available models from the GPT4All model repository. Handles potential request errors.

```python
@staticmethod
def list_models() -> list[ConfigType]:
    """
    Fetch model list from https://gpt4all.io/models/models3.json.

    Returns:
        Model list in JSON format.
    """
    resp = requests.get("https://gpt4all.io/models/models3.json")
    if resp.status_code != 200:
        raise ValueError(f"Request failed: HTTP {resp.status_code} {resp.reason}")
    return resp.json()
```

--------------------------------

### List Available GPUs

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Use this static method to retrieve a list of GPU device names available on your system. This is useful for checking hardware compatibility.

```python
@staticmethod
def list_gpus() -> list[str]:
    """
    List the names of the available GPU devices.

    Returns:
        A list of strings representing the names of the available GPU devices.
    """
    return LLModel.list_gpus()
```

--------------------------------

### Verify and Save Downloaded Model

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Verifies the MD5 hash of a partially downloaded file and moves it to its final destination. Handles potential errors during verification and file operations. Requires hashlib, tqdm, os, and Path.

```python
            if expected_md5 is not None:
                partf.seek(0)
                hsh = hashlib.md5()
                with tqdm(desc="Verifying", total=file_size, unit="iB", unit_scale=True) as bar:
                    while chunk := partf.read(block_size):
                        hsh.update(chunk)
                        bar.update(len(chunk)) 
                if hsh.hexdigest() != expected_md5.lower():
                    raise ValueError(f"Expected MD5 hash of {expected_md5!r}, got {hsh.hexdigest()!r}")
        except:
            if verbose:
                print("Cleaning up the interrupted download...", file=sys.stderr)
            try:
                os.remove(partial_path)
            except OSError:
                pass
            raise

        # flush buffers and sync the inode
        partf.flush()
        _fsync(partf)

    # move to final destination
    download_path = Path(model_path) / model_filename
    try:
        os.rename(partial_path, download_path)
    except FileExistsError:
        try:
            os.remove(partial_path)
        except OSError:
            pass
        raise

    if verbose:
        print(f"Model downloaded to {str(download_path)!r}", file=sys.stderr)
    return download_path
```

--------------------------------

### POST /chat_session

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

Initializes a context-managed chat session with the model.

```APIDOC
## POST /chat_session

### Description
Context manager to hold an inference optimized chat session with a GPT4All model.

### Method
POST

### Parameters
#### Request Body
- **system_message** (str | Literal[False] | None) - Optional - An initial instruction for the model, None to use the model default, or False to disable.
- **chat_template** (str | None) - Optional - Jinja template for the conversation, or None to use the model default.
```

--------------------------------

### Download Model Utility

Source: https://docs.gpt4all.io/gpt4all_python/ref.html

A static method to download a model file to a specified path. It takes the model filename, destination path, and optional parameters like URL and expected checksums.

```python
@staticmethod
def download_model(
    model_filename: str,
    model_path: str | os.PathLike[str],

```
Tokens:
Raw