Node Llama.cpp

https://github.com/withcatai/node-llama-cpp
Admin
Run AI models locally on your machine with pre-built bindings and Metal, CUDA, and Vulkan support,...
Tokens:88,741
Snippets:569
Trust Score:6.2
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start the project

Source: https://github.com/withcatai/node-llama-cpp/blob/master/templates/electron-typescript-react/README.md

Run this command to start the Electron app with the current setup.

```bash
npm start
```

--------------------------------

### Install dependencies with npm install

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Run this command in the project root to install all required npm dependencies.

```shell
npm install
```

--------------------------------

### Install Vulkan SDK on Ubuntu

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/Vulkan.md

Install the Vulkan SDK on Ubuntu. This script downloads the latest SDK, extracts it to /opt/vulkan-sdk, and adds the environment setup to your shell profiles.

```shell
sudo apt update
sudo apt install curl wget xz-utils

export VULKAN_VERSION="$(curl -fsSL https://vulkan.lunarg.com/sdk/latest/linux.txt)"                                                                                      

echo "Downloading Vulkan SDK version ${VULKAN_VERSION}"
curl --progress-bar "https://sdk.lunarg.com/sdk/download/${VULKAN_VERSION}/linux/vulkan_sdk.tar.xz" -o "/opt/vulkan-sdk.tar.xz"

echo "Installing Vulkan SDK to /opt/vulkan-sdk"
rm -rf "/opt/vulkan-sdk" && mkdir -p "/opt/vulkan-sdk"
tar -Jxf "/opt/vulkan-sdk.tar.xz" --strip-components=1 -C "/opt/vulkan-sdk"
rm -f "/opt/vulkan-sdk.tar.xz"

echo "Adding Vulkan SDK environment variables to shell profiles"
([ ! -f "$HOME/.bashrc" ] || grep -qxF "source /opt/vulkan-sdk/setup-env.sh" "$HOME/.bashrc") || (echo "source /opt/vulkan-sdk/setup-env.sh" >> "$HOME/.bashrc")
([ ! -f "$HOME/.zshrc" ] || grep -qxF "source /opt/vulkan-sdk/setup-env.sh" "$HOME/.zshrc") || (echo "source /opt/vulkan-sdk/setup-env.sh" >> "$HOME/.zshrc")
source /opt/vulkan-sdk/setup-env.sh
```

--------------------------------

### Download pre-built binaries after OpenMP installation

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/tips-and-tricks.md

After installing OpenMP, run this command to download pre-built binaries that will automatically use OpenMP when detected.

```shell
npx --no node-llama-cpp source download
```

--------------------------------

### Chat with a model to validate setup

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Run the chat command to test your local setup. Use `npm run dev:build` to ensure your code changes are reflected in the CLI.

```shell
npm run dev:build; node ./dist/cli/cli.js chat <path-to-a-model-file-on-your-computer>
```

--------------------------------

### Run documentation dev server

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Start the documentation development server to preview changes.

```shell
npm run docs:dev
```

--------------------------------

### Install node-llama-cpp

Source: https://github.com/withcatai/node-llama-cpp/blob/master/README.md

Install the package via npm.

```bash
npm install node-llama-cpp
```

--------------------------------

### Example output of GPU inspection with Vulkan

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/Vulkan.md

Example output from the 'npx --no node-llama-cpp inspect gpu' command when Vulkan support is available. The presence of 'Vulkan used VRAM' indicates Vulkan support is working.

```ansi
[33mVulkan:[39m [32mavailable[39m

[33mVulkan device:[39m NVIDIA RTX A6000[39m
[33mVulkan used VRAM:[39m 0% [90m(0B/47.99GB)[39m
[33mVulkan free VRAM:[39m 100% [90m(47.99GB/47.99GB)[39m

[33mCPU model:[39m Intel(R) Xeon(R) Gold 5315Y CPU @ 3.20GHz[39m
[33mUsed RAM:[39m 2.51% [90m(1.11GB/44.08GB)[39m
[33mFree RAM:[39m 97.48% [90m(42.97GB/44.08GB)[39m
```

--------------------------------

### Chatbot example

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/index.md

Loads a model, creates a context and session, then prompts the session with two questions and logs the responses.

```typescript
import {fileURLToPath} from "url";
import path from "path";
import {getLlama, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const llama = await getLlama();
const model = await llama.loadModel({
    modelPath: path.join(__dirname, "models", "Meta-Llama-3-8B-Instruct.Q4_K_M.gguf")
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});


const q1 = "Hi there, how are you?";
console.log("User: " + q1);

const a1 = await session.prompt(q1);
console.log("AI: " + a1);


const q2 = "Summarize what you said";
console.log("User: " + q2);

const a2 = await session.prompt(q2);
console.log("AI: " + a2);
```

--------------------------------

### Install OpenMP on Linux

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/tips-and-tricks.md

Install the libgomp1 package on Linux to enable OpenMP support for node-llama-cpp. This is required before building from source.

```shell
sudo apt update
sudo apt install libgomp1
```

--------------------------------

### Configure NVIDIA runtime for GPU recognition

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/docker.md

Setup steps to ensure the NVIDIA runtime is recognized by Docker or Podman.

```json
{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}
```

```shell
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list
```

--------------------------------

### npm install with --node-llama-cpp-postinstall flag

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/troubleshooting.md

Use this command to customize the postinstall behavior of node-llama-cpp during npm install. Replace 'ignoreFailedBuild' with the desired behavior option.

```shell
npm install --node-llama-cpp-postinstall=ignoreFailedBuild
```

--------------------------------

### Commit example: implement new feature

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/contributing.md

Example commit message for a new feature, using the 'feat' type, a body describing the change, and a 'Closes:' footer.

```text
feat: support more model types

This new feature adds support for importing model types 1, 2, and 3.

Closes: #22222
```

--------------------------------

### List recommended models with chat

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/index.md

Run the chat command without parameters to view a list of recommended models if you are unsure which one to start with.

```shell
npx --no node-llama-cpp chat
```

--------------------------------

### Install Termux packages for node-llama-cpp

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/troubleshooting.md

Install the required packages in Termux to allow node-llama-cpp to build binaries, since prebuilt binaries cannot be used due to the custom linker.

```bash
pkg update
pkg install nodejs git cmake clang libxml2
```

--------------------------------

### Hugging Face Model URI Examples

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/downloading-models.md

Example URIs for referencing models from Hugging Face. The first uses the optional quantization suffix, while the second specifies a particular file path. These are used with the CLI and relevant methods.

```text
hf:mradermacher/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
```

```text
hf:mradermacher/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf
```

--------------------------------

### Configure allowScripts for binaries

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/index.md

Use this command to ensure binary resolution occurs during installation by updating your package.json.

```shell
npm pkg set --json "allowScripts[node-llama-cpp]=true"
```

--------------------------------

### Get native crash stack trace on macOS with lldb

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Run node with lldb to get a stack trace for crashes in llama.cpp or bindings. After loading, type `run` (or `process launch`) to start.

```shell
lldb node -- ./node_modules/.bin/vite-node ./src/cli/cli.ts chat <path-to-a-model-file-on-your-computer>
```

--------------------------------

### Build CLI and llama.cpp with npm run dev:setup

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Builds the CLI, clones the latest llama.cpp release, builds it from source, and downloads test models. If the build fails on C++ errors, it may be due to breaking interface changes in llama.cpp; you can fix them in the llama/addon directory and open a separate PR.

```shell
npm run dev:setup
```

--------------------------------

### Get native crash stack trace on Linux with gdb

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Run node with gdb to get a stack trace for crashes in llama.cpp or bindings. After loading, type `run` to start, then `bt full` on crash.

```shell
gdb --args node ./node_modules/.bin/vite-node ./src/cli/cli.ts chat <path-to-a-model-file-on-your-computer>
```

--------------------------------

### Build documentation

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/development.md

Compile the documentation to ensure it builds correctly before sending a PR.

```shell
npm run docs:build
```

--------------------------------

### Remove gpuLayers and contextSize to fix InsufficientMemoryError

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/troubleshooting.md

Example showing the lines to remove (marked with [!code --]) when getting an InsufficientMemoryError. Remove the gpuLayers and contextSize settings to let node-llama-cpp find the optimal configuration.

```typescript
import {fileURLToPath} from "url";
import path from "path";
import {getLlama, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(
    fileURLToPath(import.meta.url)
);
const modelPath = path.join(__dirname, "my-model.gguf");
// ---cut---
const llama = await getLlama();
const model = await llama.loadModel({
    modelPath,
    gpuLayers: "max" // [!code --]
});
const context = await model.createContext({
    contextSize: 128000 // [!code --]
});
```

--------------------------------

### Run init via npm create

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/cli/init.md

Alternative way to run the init command using npm create.

```shell
npm create node-llama-cpp@latest [name]
```

--------------------------------

### Workaround for grammar and function calling with node-llama-cpp

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/grammar.md

Workaround for combining grammar and function calling: first use function calling to get a response, then prompt again with a grammar to force the output into a JSON schema. The example defines a getFruitPrice function and a grammar for an object with an itemName property, then parses the second response with grammar.parse().

```typescript
import {fileURLToPath} from "url";
import path from "path";
import {
    getLlama, LlamaChatSession, defineChatSessionFunction
} from "node-llama-cpp";

const __dirname = path.dirname(
    fileURLToPath(import.meta.url)
);

const llama = await getLlama();
const model = await llama.loadModel({
    modelPath: path.join(__dirname, "models", "Meta-Llama-3-8B-Instruct.Q4_K_M.gguf")
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const fruitPrices: Record<string, string> = {
    "apple": "$6",
    "banana": "$4"
};
const functions = {
    getFruitPrice: defineChatSessionFunction({
        description: "Get the price of a fruit",
        params: {
            type: "object",
            properties: {
                name: {
                    type: "string"
                }
            }
        },
        async handler(params) {
            const name = params.name.toLowerCase();
            if (Object.keys(fruitPrices).includes(name))
                return {
                    name: name,
                    price: fruitPrices[name]
                };

            return `Unrecognized fruit "${params.name}"`;
        }
    })
};
const grammar = await llama.createGrammarForJsonSchema({
    type: "object",
    properties: {
        itemName: {
            type: "string"
        }
    }
});

const prompt1 = "What is more expensive? An apple or a bannana?";
const res1 = await session.prompt(prompt1, {functions});
console.log("First response:", res1);

const prompt2 = "Repeat the name of the more expensive item";
const res2 = await session.prompt(prompt2, {
    grammar,
    maxTokens: context.contextSize
});
const parsedRes2 = grammar.parse(res2);

console.log("More expensive item:", parsedRes2.itemName);
```

--------------------------------

### Commit example: docs update

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/contributing.md

Example commit message for a documentation-only change, using the 'docs' type and a subject that references the 'prompt' function.

```text
docs: update documentation for the `prompt` function
```

--------------------------------

### Create and use a Prompt Completion Engine

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/chat-session.md

Demonstrates initializing a LlamaChatSession and using createPromptCompletionEngine to handle incremental prompt completions with caching.

```typescript
import {fileURLToPath} from "url";
import path from "path";
import {getLlama, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const llama = await getLlama();
const model = await llama.loadModel({
    modelPath: path.join(__dirname, "models", "Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf")
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

// ensure the model is fully loaded before continuing this demo
await session.preloadPrompt("");

const completionEngine = session.createPromptCompletionEngine({
    // 15 is used for demonstration only,
    // it's best to omit this option
    maxPreloadTokens: 15,
    // temperature: 0.8, // you can set custom generation options
    onGeneration(prompt, completion) {
        console.log(`Prompt: ${prompt} | Completion:${completion}`);
        // you should add a custom code here that checks whether
        // the existing input text equals to `prompt`, and if it does,
        // use `completion` as the completion of the input text.
        // this callback will be called multiple times
        // as the completion is being generated.
    }
});

completionEngine.complete("Hi the");

await new Promise(resolve => setTimeout(resolve, 1500));

completionEngine.complete("Hi there");
await new Promise(resolve => setTimeout(resolve, 1500));

completionEngine.complete("Hi there! How");
await new Promise(resolve => setTimeout(resolve, 1500));

// get an existing completion from the cache
// and begin/continue generating a completion for it
const cachedCompletion = completionEngine.complete("Hi there! How");
console.log("Cached completion:", cachedCompletion);
```

--------------------------------

### Commit example: fix and close issue

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/contributing.md

Example commit message for a fix that closes an issue, using the 'fix' type and a 'Closes:' footer.

```text
fix: resolve issues with model loading

Closes: #123456
```

--------------------------------

### Commit example: breaking change

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/contributing.md

Example commit message for a refactor that introduces a breaking change, using the 'refactor' type and a 'BREAKING CHANGE:' footer.

```text
refactor: refactor the function `prompt`

BREAKING CHANGE: description of breaking change in `prompt`
```

--------------------------------

### Scaffold a new project with Electron template

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/blog/v3.6-deepseek-r1.md

Use the npm create command to initialize a new project based on the latest node-llama-cpp Electron template.

```shell
npm create node-llama-cpp@latest
```

--------------------------------

### Inspect GPU support via CLI

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/index.md

Checks the current hardware configuration for GPU acceleration compatibility.

```shell
npx -y node-llama-cpp inspect gpu
```

--------------------------------

### Run container with GPU support

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/docker.md

Commands and configuration to launch a container with full GPU access.

```shell
docker run --rm -it --gpus=all my-image:tag
```

```shell
podman run --rm -it --gpus=all my-image:tag
```

```yaml
services:
  my-service:
    image: my-image:tag
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
              count: all
```

--------------------------------

### Install Windows x64 build dependencies via WinGet

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/building-from-source.md

Use WinGet to install the necessary Visual Studio 2022 Build Tools components for x64 architecture.

```shell
winget install --id Microsoft.VisualStudio.2022.BuildTools --force --override "--add Microsoft.VisualStudio.Component.VC.CMake.Project Microsoft.VisualStudio.Component.VC.CoreBuildTools Microsoft.VisualStudio.Component.VC.Tools.x86.x64 Microsoft.VisualStudio.Component.VC.ATL Microsoft.VisualStudio.Component.VC.ATLMFC Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset Microsoft.VisualStudio.Component.VC.Llvm.Clang Microsoft.VisualStudio.Component.VC.Redist.14.Latest Microsoft.Component.VC.Runtime.UCRTSDK Microsoft.VisualStudio.Component.Windows10SDK Microsoft.VisualStudio.Component.Windows10SDK.20348"
```

--------------------------------

### Install Windows on Arm build dependencies via WinGet

Source: https://github.com/withcatai/node-llama-cpp/blob/master/docs/guide/building-from-source.md

Use WinGet to install the necessary Visual Studio 2022 Build Tools components for Windows on Arm architecture.

```shell
winget install --id Microsoft.VisualStudio.2022.BuildTools --force --override "--add Microsoft.VisualStudio.Component.VC.CMake.Project Microsoft.VisualStudio.Component.VC.CoreBuildTools Microsoft.VisualStudio.Component.VC.Tools.x86.x64 Microsoft.VisualStudio.Component.VC.Tools.ARM64 Microsoft.VisualStudio.Component.VC.ATL Microsoft.VisualStudio.Component.VC.ATL.ARM64 Microsoft.VisualStudio.Component.VC.ATLMFC Microsoft.VisualStudio.Component.VC.MFC.ARM64 Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset Microsoft.VisualStudio.Component.VC.Llvm.Clang Microsoft.VisualStudio.Component.VC.Redist.14.Latest Microsoft.Component.VC.Runtime.UCRTSDK Microsoft.VisualStudio.Component.Windows10SDK Microsoft.VisualStudio.Component.Windows10SDK.20348"
```
Tokens:
Raw