### Build ggml Project

Source: https://github.com/ggml-org/ggml/blob/master/README.md

Steps to clone the ggml repository, set up a Python virtual environment, install dependencies, and build the project examples using CMake.

```bash
git clone https://github.com/ggml-org/ggml
cd ggml

# install python dependencies in a virtual environment
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# build the examples
mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8
```

--------------------------------

### Run GPT Inference Example

Source: https://github.com/ggml-org/ggml/blob/master/README.md

Commands to download a GPT-2 model and run the GPT-2 backend example with a specified model and prompt.

```bash
# run the GPT-2 small 117M model
../examples/gpt-2/download-ggml-model.sh 117M
./bin/gpt-2-backend -m models/gpt-2-117M/ggml-model.bin -p "This is an example"
```

--------------------------------

### Batched Generation Example

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/README.md

Demonstrates batched text generation using the 'gpt-2-batched' binary. This example shows loading a model, setting parameters like number of sequences (-np), prompt (-p), and maximum tokens (-n), and then displays the generated output along with performance metrics.

```bash
gpt-2-batched -np 5 -m models/gpt-2-117M/ggml-model.bin -p "Hello my name is" -n 50
```

--------------------------------

### Basic ggml-opencl Library Setup

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-opencl/CMakeLists.txt

Configures the OpenCL backend library, linking against OpenCL and setting include directories.

```cmake
find_package(OpenCL REQUIRED)
find_package(Python3 REQUIRED)

set(TARGET_NAME ggml-opencl)

ggml_add_backend_library(${TARGET_NAME}
                         ggml-opencl.cpp
                         ../../include/ggml-opencl.h)
target_link_libraries(${TARGET_NAME} PRIVATE ${OpenCL_LIBRARIES})
target_include_directories(${TARGET_NAME} PRIVATE ${OpenCL_INCLUDE_DIRS})
```

--------------------------------

### Install CMake Package Configuration Files

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Installs the generated ggml-config.cmake and ggml-version.cmake files to the appropriate CMake package directory.

```cmake
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake
              ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml)
```

--------------------------------

### Install Metal Library

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-metal/CMakeLists.txt

Installs the compiled Metal library to the binary directory. This installation step is conditional on GGML_METAL_EMBED_LIBRARY not being set.

```cmake
install(
    FILES ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/default.metallib
    DESTINATION ${CMAKE_INSTALL_BINDIR}
)
```

--------------------------------

### Install HTP Library

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-hexagon/htp/CMakeLists.txt

Installs the built HTP shared library. This makes the library available for use by other projects or for deployment.

```cmake
install(TARGETS ${HTP_LIB})

```

--------------------------------

### Run Web Demo HTTP Server

Source: https://github.com/ggml-org/ggml/blob/master/examples/mnist/README.md

Start a simple HTTP server using Python 3 to serve the compiled web demo files. This command serves the contents of the 'web' directory, making the demo accessible via a local URL.

```bash
python3 examples/mnist/server.py
```

--------------------------------

### Apply MSVC Configuration to Example Targets

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Conditionally applies MSVC configurations to various example targets if GGML_BUILD_EXAMPLES is enabled.

```cmake
if (GGML_BUILD_EXAMPLES)
        configure_msvc_target(common-ggml)
        configure_msvc_target(common)

        configure_msvc_target(mnist-common)
        configure_msvc_target(mnist-eval)
        configure_msvc_target(mnist-train)

        configure_msvc_target(gpt-2-ctx)
        configure_msvc_target(gpt-2-alloc)
        configure_msvc_target(gpt-2-backend)
        configure_msvc_target(gpt-2-sched)
        configure_msvc_target(gpt-2-quantize)
        configure_msvc_target(gpt-2-batched)

        configure_msvc_target(gpt-j)
        configure_msvc_target(gpt-j-quantize)

        configure_msvc_target(magika)
        configure_msvc_target(yolov3-tiny)
        configure_msvc_target(sam)

        configure_msvc_target(simple-ctx)
        configure_msvc_target(simple-backend)
    endif()
```

--------------------------------

### Basic CMake Project Setup

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-hexagon/htp/CMakeLists.txt

Initializes the CMake build system and defines the project name and supported languages. Requires CMake version 3.22.2 or higher.

```cmake
cmake_minimum_required(VERSION 3.22.2)
project(ggml-htp C CXX ASM)
```

--------------------------------

### Install Hexagon Skels at Runtime

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-hexagon/CMakeLists.txt

This command installs the built Hexagon skeleton libraries to the appropriate runtime location. It ensures that the necessary shared objects are available when the application is executed.

```cmake
install(FILES ${HTP_SKELS} TYPE LIB)
```

--------------------------------

### Configure CANN Installation Directory

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-cann/CMakeLists.txt

Sets the CANN installation directory based on environment variables or aborts if not found. Ensure the CANN environment is sourced before building.

```cmake
if ("cann${CANN_INSTALL_DIR}" STREQUAL "cann" AND DEFINED ENV{ASCEND_TOOLKIT_HOME})
    set(CANN_INSTALL_DIR $ENV{ASCEND_TOOLKIT_HOME})
    message(STATUS "CANN: updated CANN_INSTALL_DIR from ASCEND_TOOLKIT_HOME=$ENV{ASCEND_TOOLKIT_HOME}")
endif()
```

--------------------------------

### GPT-2 Inference Sample Output

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/README.md

Example output from running the GPT-2 inference program without specific arguments, showing model loading details and generated text.

```bash
$ ./bin/gpt-2
gpt2_model_load: loading model from 'models/gpt-2-117M/ggml-model.bin'
gpt2_model_load: n_vocab = 50257
gpt2_model_load: n_ctx   = 1024
gpt2_model_load: n_embd  = 768
gpt2_model_load: n_head  = 12
gpt2_model_load: n_layer = 12
gpt2_model_load: f16     = 1
gpt2_model_load: ggml ctx size = 311.12 MB
gpt2_model_load: memory size =    72.00 MB, n_mem = 12288
gpt2_model_load: model size  =   239.08 MB
main: number of tokens in prompt = 1

So this is going to be the end of the line for us.

If the Dolphins continue to do their business, it's possible that the team could make a bid to bring in new defensive coordinator Scott Linehan.

Linehan's job is a little daunting, but he's a great coach and an excellent coach. I don't believe we're going to make the playoffs.

We're going to have to work hard to keep our heads down and get ready to go.<|endoftext|>

main: mem per token =  2048612 bytes
main:     load time =   106.32 ms
main:   sample time =     7.10 ms
main:  predict time =   506.40 ms / 5.06 ms per token
main:    total time =   629.84 ms

```

--------------------------------

### Run GPT-J Inference with Prompt from Pipe

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-j/README.md

This example demonstrates how to pipe a text prompt from a file to the GPT-J inference tool. Ensure the ggml-model.bin file is available in the specified path.

```bash
echo "This is an example" > prompt.txt
cat prompt.txt | ./bin/gpt-j -m models/gpt-j-6B/ggml-model.bin
```

--------------------------------

### Configure Package Configuration File

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Generates the ggml-config.cmake file from a template, specifying installation destination and path variables.

```cmake
configure_package_config_file(
        ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in
        ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake
    INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml
    PATH_VARS GGML_INCLUDE_INSTALL_DIR
              GGML_LIB_INSTALL_DIR
              GGML_BIN_INSTALL_DIR)
```

--------------------------------

### Install Metal Kernels

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-metal/CMakeLists.txt

Installs the Metal kernel source file to the binary directory during the build process. This rule is applied only when GGML_METAL_EMBED_LIBRARY is not enabled.

```cmake
install(
    FILES src/ggml-metal/ggml-metal.metal
    PERMISSIONS
        OWNER_READ
        OWNER_WRITE
        GROUP_READ
        WORLD_READ
    DESTINATION ${CMAKE_INSTALL_BINDIR})

```

--------------------------------

### Run SAM Inference with ggml

Source: https://github.com/ggml-org/ggml/blob/master/examples/sam/README.md

This command downloads the ViT-B SAM model, converts it to ggml format, and then runs inference on an example image. Ensure Python is set up and examples are built.

```bash
# Download PTH model
wget -P examples/sam/ https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth

# Convert PTH model to ggml
python examples/sam/convert-pth-to-ggml.py examples/sam/sam_vit_b_01ec64.pth examples/sam/ 1

# run inference
./bin/sam -t 16 -i ../examples/sam/example.jpg -m ../examples/sam/ggml-model-f16.bin
```

--------------------------------

### Configure OpenVINO and OpenCL Dependencies

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-openvino/CMakeLists.txt

Finds and includes necessary OpenVINO and OpenCL CMake packages. Ensure OpenVINO is installed and discoverable by CMake.

```cmake
find_package(OpenVINO REQUIRED)
find_package(OpenCL REQUIRED)

include("${OpenVINO_DIR}/../3rdparty/tbb/lib/cmake/TBB/TBBConfig.cmake")
```

--------------------------------

### Download YOLOv3-tiny Weights

Source: https://github.com/ggml-org/ggml/blob/master/examples/yolo/README.md

Use wget to download the pretrained YOLOv3-tiny model weights. Verify the download using sha1sum.

```bash
$ wget https://pjreddie.com/media/files/yolov3-tiny.weights
$ sha1sum yolov3-tiny.weights 
40f3c11883bef62fd850213bc14266632ed4414f  yolov3-tiny.weights
```

--------------------------------

### GPT-2 Backend Executable

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/CMakeLists.txt

Sets up the 'gpt-2-backend' executable using 'main-backend.cpp' and links it with the 'ggml', 'common', and 'common-ggml' libraries.

```cmake
set(TEST_TARGET gpt-2-backend)
add_executable(${TEST_TARGET} main-backend.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml common common-ggml)
```

--------------------------------

### Basic GGML Tensor Operations in Python

Source: https://github.com/ggml-org/ggml/blob/master/examples/python/README.md

Demonstrates initializing a GGML context, creating tensors, performing addition, building a computation graph, and copying data. Ensure GGML_LIBRARY is set correctly.

```python
# Make sure libllama.so is in your [DY]LD_LIBRARY_PATH, or set GGML_LIBRARY=.../libggml_shared.so

from ggml import lib, ffi
from ggml.utils import init, copy, numpy
import numpy as np

ctx = init(mem_size=12*1024*1024)
n = 256
n_threads = 4

a = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_Q5_K, n)
b = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_F32, n) # Can't both be quantized
sum = lib.ggml_add(ctx, a, b) # all zeroes for now. Will be quantized too!

gf = ffi.new('struct ggml_cgraph*')
lib.ggml_build_forward_expand(gf, sum)

copy(np.array([i for i in range(n)], np.float32), a)
copy(np.array([i*100 for i in range(n)], np.float32), b)

lib.ggml_graph_compute_with_ctx(ctx, gf, n_threads)

print(numpy(a, allow_copy=True))
#  0.    1.0439453   2.0878906   3.131836    4.1757812   5.2197266. ...
print(numpy(b))
#  0.  100.        200.        300.        400.        500.         ...
print(numpy(sum, allow_copy=True))
#  0.  105.4375    210.875     316.3125    421.75      527.1875     ...
```

--------------------------------

### Set GGML Installation Version and Paths

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Defines the version for the GGML package and sets cache variables for the installation directories of include files, library files, and binary files.

```cmake
set(GGML_INSTALL_VERSION ${GGML_VERSION})
set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header  files")
set(GGML_LIB_INSTALL_DIR     ${CMAKE_INSTALL_LIBDIR}     CACHE PATH "Location of library files")
set(GGML_BIN_INSTALL_DIR     ${CMAKE_INSTALL_BINDIR}     CACHE PATH "Location of binary  files")
```

--------------------------------

### Build and Run SAM Inference

Source: https://github.com/ggml-org/ggml/blob/master/examples/sam/README.md

This command sequence first builds the SAM executable using make and then runs inference on an image, displaying the build process and inference time. The output shows model loading details and mask processing.

```bash
$ ▶ make -j sam && time ./bin/sam -t 8 -i img.jpg
[ 28%] Built target common
[ 71%] Built target ggml
[100%] Built target sam
main: seed = 1693224265
main: loaded image 'img.jpg' (680 x 453)
sam_image_preprocess: scale = 0.664062
main: preprocessed image (1024 x 1024)
sam_model_load: loading model from 'models/sam-vit-b/ggml-model-f16.bin' - please wait ...
sam_model_load: n_enc_state      = 768
sam_model_load: n_enc_layer      = 12
sam_model_load: n_enc_head       = 12
sam_model_load: n_enc_out_chans  = 256
sam_model_load: n_pt_embd        = 4
sam_model_load: ftype            = 1
sam_model_load: qntvr            = 0
operator(): ggml ctx size = 202.32 MB
sam_model_load: ...................................... done
sam_model_load: model size =   185.05 MB / num tensors = 304
embd_img
dims: 64 64 256 1 f32
First & Last 10 elements:
-0.05117 -0.06408 -0.07154 -0.06991 -0.07212 -0.07690 -0.07508 -0.07281 -0.07383 -0.06779
0.01589 0.01775 0.02250 0.01675 0.01766 0.01661 0.01811 0.02051 0.02103 0.03382
sum:  12736.272313

Skipping mask 0 with iou 0.705935 below threshold 0.880000
Skipping mask 1 with iou 0.762136 below threshold 0.880000
Mask 2: iou = 0.947081, stability_score = 0.955437, bbox (371, 436), (144, 168)


main:     load time =    51.28 ms
main:    total time =  2047.49 ms

real	0m2.068s
user	0m16.343s
sys	0m0.214s
```

--------------------------------

### Configure test-quantize-fns Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-quantize-fns' executable, links it with GGML, and registers it as a test. Includes LLVM profiling configuration.

```cmake
set(TEST_TARGET test-quantize-fns)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
set_property(TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=${TEST_TARGET}.profraw")
```

--------------------------------

### Configure test-conv-transpose-1d Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-conv-transpose-1d' executable, links it with GGML, and registers it as a test.

```cmake
set(TEST_TARGET test-conv-transpose-1d)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
```

--------------------------------

### Configure 'sam-quantize' Executable Build (Commented Out)

Source: https://github.com/ggml-org/ggml/blob/master/examples/sam/CMakeLists.txt

This commented-out section shows how to configure the 'sam-quantize' executable using 'quantize.cpp' and linking against 'ggml' and 'common'. It is currently disabled.

```cmake
#set(TEST_TARGET sam-quantize)
#add_executable(${TEST_TARGET} quantize.cpp)
#target_link_libraries(${TEST_TARGET} PRIVATE ggml common)
```

--------------------------------

### Find Metal Frameworks

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-metal/CMakeLists.txt

Locates required macOS frameworks for Metal support. Ensure these frameworks are installed on your system.

```cmake
find_library(FOUNDATION_LIBRARY Foundation REQUIRED)
find_library(METAL_FRAMEWORK    Metal      REQUIRED)
find_library(METALKIT_FRAMEWORK MetalKit   REQUIRED)
```

--------------------------------

### Configure test-conv1d Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-conv1d' executable, links it with GGML, and registers it as a test.

```cmake
set(TEST_TARGET test-conv1d)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
```

--------------------------------

### Get Git Commit Hash

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Executes the Git command to retrieve the short commit hash of the current HEAD. This is used to identify the build.

```cmake
execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
    OUTPUT_VARIABLE GGML_BUILD_COMMIT
    OUTPUT_STRIP_TRAILING_WHITESPACE
    ERROR_QUIET
)
```

--------------------------------

### Configure test-backend-ops Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-backend-ops' executable, links it with the GGML library and Threads, and registers it as a test. It also configures LLVM profiling.

```cmake
set(TEST_TARGET test-backend-ops)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml Threads::Threads)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
set_property(TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=${TEST_TARGET}.profraw")
```

--------------------------------

### HIP Version Check

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-hip/CMakeLists.txt

Ensures that the installed ROCm/HIP version is at least V6.1, which is required for certain features. If the version is less than 6.1, the build will fail.

```cmake
if (${hip_VERSION} VERSION_LESS 6.1)
    message(FATAL_ERROR "At least ROCM/HIP V6.1 is required")
endif()

message(STATUS "HIP and hipBLAS found")
```

--------------------------------

### Configure test-rel-pos Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-rel-pos' executable, links it with GGML, and registers it as a test.

```cmake
set(TEST_TARGET test-rel-pos)
add_executable(${TEST_TARGET} ${TEST_TARGET}.c)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
```

--------------------------------

### Configure test-pad-reflect-1d Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Defines the 'test-pad-reflect-1d' executable, links it with GGML, and registers it as a test.

```cmake
set(TEST_TARGET test-pad-reflect-1d)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
```

--------------------------------

### CMakeLists.txt for GGML Project

Source: https://github.com/ggml-org/ggml/blob/master/examples/test-cmake/CMakeLists.txt

This CMakeLists.txt file configures a C++ project to use the GGML library. It requires CMake version 3.14 or higher and C++17 standard. Ensure the GGML library is installed and discoverable by CMake's find_package command.

```cmake
cmake_minimum_required(VERSION 3.14)
project(ggml-simple)

set(CMAKE_CXX_STANDARD 17)

find_package(ggml CONFIG REQUIRED)

set(TEST_TARGET test-cmake)
add_executable(test-cmake test-cmake.cpp)
target_link_libraries(test-cmake PRIVATE ggml::ggml)
```

--------------------------------

### Append Compiler Flags for CUDA

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-cuda/CMakeLists.txt

This section appends specific compiler flags to the CUDA build process. It adds '-Wno-pedantic' for non-MSVC builds and '/Zc:preprocessor' for MSVC builds starting from CCCL 3.2, ensuring compliance with preprocessor standards.

```cmake
list(APPEND CUDA_CXX_FLAGS ${CXX_FLAGS} ${GF_CXX_FLAGS})  # This is passed to -Xcompiler later
    endif()

    if (NOT MSVC)
        list(APPEND CUDA_CXX_FLAGS -Wno-pedantic)
    else()
        # CCCL 3.2 onwards will require a cpp-standard-compliant preprocessor for MSVC
        # https://github.com/NVIDIA/cccl/pull/6827
        list(APPEND CUDA_CXX_FLAGS /Zc:preprocessor)
    endif()

    list(JOIN   CUDA_CXX_FLAGS " " CUDA_CXX_FLAGS_JOINED)  # pass host compiler flags as a single argument

    if (NOT CUDA_CXX_FLAGS_JOINED STREQUAL "")
        list(APPEND CUDA_FLAGS -Xcompiler ${CUDA_CXX_FLAGS_JOINED})
    endif()

    target_compile_options(ggml-cuda PRIVATE "<$<COMPILE_LANGUAGE:CUDA>:${CUDA_FLAGS}>")
else()
    message(FATAL_ERROR "CUDA Toolkit not found")
endif()
```

--------------------------------

### GPT-2 Inference Help Output

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/README.md

Displays the command-line options for the GPT-2 inference executable. Use this to understand available parameters for controlling generation.

```bash
$ ./bin/gpt-2 -h
usage: ./bin/gpt-2 [options]

options:
  -h, --help            show this help message and exit
  -s SEED, --seed SEED  RNG seed (default: -1)
  -t N, --threads N     number of threads to use during computation (default: 8)
  -p PROMPT, --prompt PROMPT
                        prompt to start generation with (default: random)
  -n N, --n_predict N   number of tokens to predict (default: 200)
  --top_k N             top-k sampling (default: 40)
  --top_p N             top-p sampling (default: 0.9)
  --temp N              temperature (1.0)
  -b N, --batch_size N  batch size for prompt processing (default: 8)
  -m FNAME, --model FNAME
                        model path (default: models/gpt-2-117M/ggml-model.bin)

```

--------------------------------

### Configure POSIX.1-2008 for OpenBSD

Source: https://github.com/ggml-org/ggml/blob/master/src/CMakeLists.txt

Sets the _XOPEN_SOURCE to 700 for OpenBSD to comply with POSIX.1-2008 standards.

```cmake
if (CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
    add_compile_definitions(_XOPEN_SOURCE=700)
elseif (CMAKE_SYSTEM_NAME MATCHES "AIX")
    # Don't define _XOPEN_SOURCE.  We need _ALL_SOURCE, which is the default,
    # in order to define _SC_PHYS_PAGES.
else()
    add_compile_definitions(_XOPEN_SOURCE=600)
endif()
```

--------------------------------

### Compile GGML Evaluation Code with Emscripten

Source: https://github.com/ggml-org/ggml/blob/master/examples/mnist/README.md

Compile the GGML evaluation code to WebAssembly using Emscripten. Ensure you are in the root directory of the project and have prepared the model and test set files in the specified location. This command sets up the build environment and compiles the 'mnist' example.

```bash
cd ../..
mkdir -p build-em
emcmake cmake .. -DGGML_BUILD_EXAMPLES=ON \
    -DCMAKE_C_FLAGS="-pthread -matomics -mbulk-memory" \
    -DCMAKE_CXX_FLAGS="-pthread -matomics -mbulk-memory"
make mnist
```

--------------------------------

### Configure test-quantize-perf Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Configures the 'test-quantize-perf' executable, linking it to GGML and setting it up as a test with LLVM profiling.

```cmake
set(TEST_TARGET test-quantize-perf)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
set_property(TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=${TEST_TARGET}.profraw")
```

--------------------------------

### Train Fully Connected Model on CPU (GGML)

Source: https://github.com/ggml-org/ggml/blob/master/examples/mnist/README.md

Train a fully connected model directly on the CPU using the GGML training binary. Requires the model name, output GGUF file, and training dataset paths.

```bash
$ ../../build/bin/mnist-train mnist-fc mnist-fc-f32.gguf data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-labels-idx1-ubyte
```

--------------------------------

### Configure test-pool Executable with Stack Option

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-pool' executable, links it with GGML, and registers it as a test. Includes a conditional stack size option for MSVC.

```cmake
set(TEST_TARGET test-pool)
add_executable(${TEST_TARGET} ${TEST_TARGET}.c)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
if (MSVC)
    target_link_options(${TEST_TARGET} PRIVATE "/STACK:8388608") # 8MB
endif()
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
set_property(TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=${TEST_TARGET}.profraw")
```

--------------------------------

### Configure Windows SDK Paths for Signing

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-hexagon/CMakeLists.txt

This section configures paths to the Windows SDK binaries, specifically for ARM64 and x86 architectures, when building on a Windows system and code signing is enabled. It helps locate necessary signing tools.

```cmake
if (CMAKE_SYSTEM_NAME MATCHES Windows AND GGML_HEXAGON_HTP_CERT)
    file(TO_CMAKE_PATH "$ENV{WINDOWS_SDK_BIN}/arm64"      WINSDK_BIN0_ARM64)
    file(TO_CMAKE_PATH "$ENV{WINDOWS_SDK_BIN}/x86"        WINSDK_BIN0_X86)
    file(TO_CMAKE_PATH "$ENV{WindowsSdkVerBinPath}/arm64" WINSDK_BIN1_ARM64)
    file(TO_CMAKE_PATH "$ENV{WindowsSdkVerBinPath}/x86"   WINSDK_BIN1_X86)

    set(WINSDK_PATHS ${WINSDK_BIN0_ARM64} ${WINSDK_BIN0_X86} ${WINSDK_BIN1_ARM64} ${WINSDK_BIN1_X86})

    find_program(INF2CAT  NAMES inf2cat.exe  PATHS ${WINSDK_PATHS} REQUIRED)
    find_program(SIGNTOOL NAMES signtool.exe PATHS ${WINSDK_PATHS} REQUIRED)

    message(STATUS "hexagon: using ${GGML_HEXAGON_HTP_CERT} to sign libggml-htp skels")

    set(LIBGGML_HTP_CAT ${CMAKE_CURRENT_BINARY_DIR}/libggml-htp.cat)
    add_custom_target(libggml-htp-cat
        BYPRODUCTS ${LIBGGML_HTP_CAT}
        DEPENDS libggml-htp.inf ${HTP_SKELS}
        COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/libggml-htp.inf ${CMAKE_CURRENT_BINARY_DIR}
        COMMAND ${INF2CAT} /driver:${CMAKE_CURRENT_BINARY_DIR} /os:10_25H2_ARM64
        COMMAND ${SIGNTOOL} sign /fd sha256 /f ${GGML_HEXAGON_HTP_CERT} ${LIBGGML_HTP_CAT}
        COMMENT "generating and signing libggml-htp.cat file"
        VERBATIM
    )

    add_dependencies(${TARGET_NAME} libggml-htp-cat)
    install(FILES ${LIBGGML_HTP_CAT} TYPE LIB)
endif()
```

--------------------------------

### Configure VirtGPU Backend Library with CMake

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-virtgpu/backend/CMakeLists.txt

Use this CMake script to add the ggml-virtgpu-backend library. Ensure you have the necessary source files and headers in the specified paths.

```cmake
cmake_minimum_required(VERSION 3.19)
cmake_policy(SET CMP0114 NEW)

message(STATUS "Enable the VirtGPU/Virglrenderer backend library")

ggml_add_backend_library(ggml-virtgpu-backend
                         backend.cpp
                         backend-dispatched.cpp
                         backend-dispatched-backend.cpp
                         backend-dispatched-device.cpp
                         backend-dispatched-buffer.cpp
                         backend-dispatched-buffer-type.cpp
                         shared/api_remoting.h
                         shared/apir_backend.h
                         shared/apir_cs.h
                         apir_cs_ggml-rpc-back.cpp)

target_compile_options(ggml-virtgpu-backend PRIVATE -std=c++20)

# Add include directory for ggml-backend-impl.h and other core headers
target_include_directories(ggml-virtgpu-backend PRIVATE ../..)

```

--------------------------------

### Write Basic Package Version File

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Creates a ggml-version.cmake file to store the package version, used for version management.

```cmake
write_basic_package_version_file(
        ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake
    VERSION ${GGML_INSTALL_VERSION}
    COMPATIBILITY SameMajorVersion)
```

--------------------------------

### Handle Unsupported s390x Target OS

Source: https://github.com/ggml-org/ggml/blob/master/src/CMakeLists.txt

Logs a fatal error if the target OS for s390x architecture is not supported.

```cmake
message(FATAL_ERROR "Unsupported s390x target OS: ${CMAKE_SYSTEM_NAME}")
```

--------------------------------

### Run Downloaded GGML Model

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/README.md

Execute the downloaded GGML model using the provided binary and model path. This command is used after successfully downloading the model.

```bash
./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "This is an example"
```

--------------------------------

### Configure MNIST Training Executable

Source: https://github.com/ggml-org/ggml/blob/master/examples/mnist/CMakeLists.txt

Builds an executable for MNIST training. Links against ggml, common, and the mnist-common library.

```cmake
set(TEST_TARGET mnist-train)
add_executable(${TEST_TARGET} mnist-train.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml common mnist-common)
```

--------------------------------

### Configure test-roll Executable

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-roll' executable, links it with GGML, and registers it as a test.

```cmake
set(TEST_TARGET test-roll)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
```

--------------------------------

### Configure test-timestep_embedding Executable with Stack Option

Source: https://github.com/ggml-org/ggml/blob/master/tests/CMakeLists.txt

Sets up the 'test-timestep_embedding' executable, links it with GGML, and registers it as a test. Includes a conditional stack size option for MSVC.

```cmake
set(TEST_TARGET test-timestep_embedding)
add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
if (MSVC)
    target_link_options(${TEST_TARGET} PRIVATE "/STACK:8388608") # 8MB
endif()
add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}>)
set_property(TEST ${TEST_TARGET} PROPERTY ENVIRONMENT "LLVM_PROFILE_FILE=${TEST_TARGET}.profraw")
```

--------------------------------

### Enable Ccache

Source: https://github.com/ggml-org/ggml/blob/master/CMakeLists.txt

Option to use ccache for faster compilation if it is available. Defaults to ON.

```cmake
option(GGML_CCACHE "ggml: use ccache if available"                   ON)
```

--------------------------------

### Add CPU Backend

Source: https://github.com/ggml-org/ggml/blob/master/src/CMakeLists.txt

Adds the main CPU backend to the build system. This is a foundational step before defining specific variants.

```cmake
ggml_add_backend(CPU)

```

--------------------------------

### Build simple-backend Executable with GGML Library

Source: https://github.com/ggml-org/ggml/blob/master/examples/simple/CMakeLists.txt

Configures a CMake build to create an executable named 'simple-backend' from 'simple-backend.cpp' and links it against the GGML library.

```cmake
set(TEST_TARGET simple-backend)
add_executable(${TEST_TARGET} simple-backend.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml)
```

--------------------------------

### Run GPT-J Inference with Custom Parameters

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-j/README.md

This command demonstrates GPT-J inference with custom parameters for token count (-n 500) and thread count (-t 8), along with a different prompt. The output logs show model loading and inference details for the specified configuration.

```bash
time ./bin/gpt-j -n 500 -t 8 -p "Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?
"

gptj_model_load: loading model from 'models/gpt-j-6B/ggml-model.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 1
gptj_model_load: ggml ctx size = 13334.86 MB
gptj_model_load: memory_size =  1792.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size = 11542.79 MB / num tensors = 285
main: number of tokens in prompt = 24

Ask HN: Inherited the worst code and tech team I have ever seen. How to fix it?

I've inherited a team with some very strange and un-documented practices, one of them is that they use an old custom
application with a very slow tech stack written in Python that the team doesn't want to touch but also doesn't want to
throw away as it has some "legacy" code in it.

The problem is, the tech stack is very very slow.

They have a single web server on a VM that is slow.
The server is a little bit busy (not very busy though) and they have a lot of processes (30+ that are constantly being
spawned by the application)
They have an application that is single threaded and was written in Python and the team don't want to touch this, and
the application is very slow.

My task as a new member of the team is to fix this.

I'm a senior dev on the team (3 years on the project) and have been told that I will take the lead on this task. I know
next to nothing about Python. So here is what I have so far.

What I have done is I've been trying to debug the processes with the "ps" command. This way I can see what is running
and where. From what I see, the application spawns 10 processes a minute and some of them are used for nothing.

I have also started to look for the code. The application source is not in GitHub or any other repository, it is only on
our internal GitLab.

What I've found so far:

The application uses a custom SQLAlchemy implementation to interact with the data. I've looked at the source, it looks

```

--------------------------------

### Add Backend Function

Source: https://github.com/ggml-org/ggml/blob/master/src/CMakeLists.txt

Includes a backend subdirectory and configures compile definitions based on whether dynamic loading is enabled.

```cmake
function(ggml_add_backend backend)
    string(TOUPPER "GGML_${backend}" backend_id)
    if (${backend_id})
        string(TOLOWER "ggml-${backend}" backend_target)
        add_subdirectory(${backend_target})
        message(STATUS "Including ${backend} backend")
        if (NOT GGML_BACKEND_DL)
            string(TOUPPER "GGML_USE_${backend}" backend_use)
            target_compile_definitions(ggml PUBLIC ${backend_use})
        endif ()
    endif ()
endfunction()
```

--------------------------------

### Configure VirtGPU/Virglrenderer Backend

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-virtgpu/CMakeLists.txt

This CMake code block configures the VirtGPU/Virglrenderer backend. It checks for libdrm, defines GGML_USE_VIRTGPU_FRONTEND if not using dynamic loading, and adds the ggml-virtgpu library with its source files, include directories, and link libraries. It also ensures the venus_hw.h header is downloaded before building.

```cmake
if (NOT GGML_VIRTGPU_BACKEND STREQUAL "ONLY")
    message(STATUS "Enable the VirtGPU/Virglrenderer API Remoting frontend library")

    find_package(PkgConfig REQUIRED)
    pkg_check_modules(DRM REQUIRED libdrm)
    if (NOT GGML_BACKEND_DL)
      # cannot simply use USE_VIRTGPU, as in the 'else()' case the
      # frontend isn't compiled
      target_compile_definitions(ggml PUBLIC "GGML_USE_VIRTGPU_FRONTEND")
    endif()

    ggml_add_backend_library(ggml-virtgpu
                             ggml-backend-buffer.cpp
                             ggml-backend.cpp
                             ggml-backend-device.cpp
                             ggml-backend-reg.cpp
                             ggml-backend-buffer-type.cpp
                             virtgpu-apir.h
                             virtgpu-forward.gen.h
                             virtgpu.cpp
                             virtgpu-shm.cpp
                             virtgpu-utils.cpp
                             virtgpu-forward-device.cpp
                             virtgpu-forward-buffer-type.cpp
                             virtgpu-forward-buffer.cpp
                             virtgpu-forward-backend.cpp
                             virtgpu-forward-impl.h
                             apir_cs_ggml-rpc-front.cpp
                             ../../include/ggml-virtgpu.h)

    target_include_directories(ggml-virtgpu PUBLIC /usr/include/libdrm/)

    target_link_libraries(ggml-virtgpu PUBLIC ${DRM_LIBRARIES})
    target_include_directories(ggml-virtgpu PUBLIC ${DRM_INCLUDE_DIRS})
    target_compile_options(ggml-virtgpu PUBLIC ${DRM_CFLAGS_OTHER})

    target_include_directories(ggml-virtgpu PUBLIC ./include)
    target_include_directories(ggml-virtgpu PRIVATE ${CMAKE_CURRENT_BINARY_DIR})

    # Ensure venus_hw.h is downloaded before building ggml-virtgpu
    add_dependencies(ggml-virtgpu venus_hw_header)

    target_compile_options(ggml-virtgpu PRIVATE -std=c++20)
else()
    message(STATUS "Not building the VirtGPU/Virglrenderer API Remoting frontend library")
endif()

if (NOT GGML_VIRTGPU_BACKEND STREQUAL "OFF")
    add_subdirectory("backend")
endif()
```

--------------------------------

### Configure Yolov3-tiny Executable

Source: https://github.com/ggml-org/ggml/blob/master/examples/yolo/CMakeLists.txt

Use this CMake configuration to build the yolov3-tiny executable. It links against the 'ggml' and 'common' libraries.

```cmake
set(TEST_TARGET yolov3-tiny)
add_executable(${TEST_TARGET} yolov3-tiny.cpp yolo-image.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml common)
```

--------------------------------

### s390x Architecture Flags

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-cpu/CMakeLists.txt

Adds the s390x architecture specific quantizations source file.

```cmake
elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
        message(STATUS "s390x detected")
        list(APPEND GGML_CPU_SOURCES
            ggml-cpu/arch/s390/quants.c)

```

--------------------------------

### Evaluate Model on CPU (GGML)

Source: https://github.com/ggml-org/ggml/blob/master/examples/mnist/README.md

Evaluate a trained fully connected model on the CPU using the GGML evaluation binary. Requires the GGUF model file and MNIST dataset paths.

```bash
$ ../../build/bin/mnist-eval mnist-fc-f32.gguf data/MNIST/raw/t10k-images-idx3-ubyte data/MNIST/raw/t10k-labels-idx1-ubyte
```

--------------------------------

### Download GPT-2 Model Files

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-2/README.md

Bash script to download original GPT-2 model files. Ensure you are in the 'ggml/build' directory before running.

```bash
cd ggml/build
../examples/gpt-2/download-model.sh 117M

Downloading model 117M ...
models/gpt-2-117M/checkpoint                      100%[=============================>]      77  --.-KB/s    in 0s
models/gpt-2-117M/encoder.json                    100%[=============================>]   1018K  1.20MB/s    in 0.8s
models/gpt-2-117M/hparams.json                    100%[=============================>]      90  --.-KB/s    in 0s
models/gpt-2-117M/model.ckpt.data-00000-of-00001  100%[=============================>] 474.70M  1.21MB/s    in 8m 39s
models/gpt-2-117M/model.ckpt.index                100%[=============================>]   5.09K  --.-KB/s    in 0s
models/gpt-2-117M/model.ckpt.meta                 100%[=============================>] 460.11K   806KB/s    in 0.6s
models/gpt-2-117M/vocab.bpe                       100%[=============================>] 445.62K   799KB/s    in 0.6s
Done! Model '117M' saved in 'models/gpt-2-117M/'

```

--------------------------------

### Configure Static Linking and Profiling Flags

Source: https://github.com/ggml-org/ggml/blob/master/src/CMakeLists.txt

Sets up linker suffixes for static libraries on Unix-like systems and adds the -static flag. It also enables profiling with the -pg flag if GGML_GPROF is defined.

```cmake
if (NOT MSVC)
    if (GGML_STATIC)
        if (UNIX AND NOT APPLE)
            set(CMAKE_FIND_LIBRARY_SUFFIXES ".a;.so")
        endif()
        add_link_options(-static)
        if (MINGW)
            add_link_options(-static-libgcc -static-libstdc++)
        endif()
    endif()
    if (GGML_GPROF)
        add_compile_options(-pg)
    endif()
endif()
```

--------------------------------

### Run GPT-J Inference with Default Parameters

Source: https://github.com/ggml-org/ggml/blob/master/examples/gpt-j/README.md

This command executes the GPT-J model inference with a specific prompt. It includes detailed output logs showing model loading, memory allocation, and inference timing. Observe the 'predict time' for token generation speed.

```bash
$ time ./bin/gpt-j -p "int main(int argc, char ** argv) {"

gptj_model_load: loading model from 'models/gpt-j-6B/ggml-model.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx   = 2048
gptj_model_load: n_embd  = 4096
gptj_model_load: n_head  = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot   = 64
gptj_model_load: f16     = 1
gptj_model_load: ggml ctx size = 13334.86 MB
gptj_model_load: memory_size =  1792.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size = 11542.79 MB / num tensors = 285
main: number of tokens in prompt = 13

int main(int argc, char ** argv) {
    (void)argc;
    (void)argv;

    {
        struct sockaddr_in addr;
        int addrlen;
        char * ip = "192.168.1.4";
        int i;

        if ( (addrlen = sizeof(addr)) == -1 )
            return -1;

        for (i = 0; i < 10; ++i) {
            addr.sin_family = AF_INET;
            addr.sin_addr.s_addr = inet_addr(ip);

main: mem per token = 16430420 bytes
main:     load time =  6211.48 ms
main:   sample time =    13.74 ms
main:  predict time = 26420.34 ms / 124.62 ms per token
main:    total time = 33035.37 ms

real	0m33.171s
user	3m32.269s
sys	0m3.686s

$
```

--------------------------------

### Link OpenVINO, TBB, and OpenCL Libraries

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-openvino/CMakeLists.txt

Links the ggml-openvino target with the required OpenVINO runtime, TBB, and OpenCL libraries. These are essential for OpenVINO functionality.

```cmake
target_link_libraries(ggml-openvino PRIVATE openvino::runtime TBB::tbb OpenCL::OpenCL)
```

--------------------------------

### Configure 'sam' Executable Build

Source: https://github.com/ggml-org/ggml/blob/master/examples/sam/CMakeLists.txt

Defines the 'sam' executable using 'sam.cpp' and links it against the 'ggml' and 'common' libraries. This is used for building the 'sam' tool.

```cmake
set(TEST_TARGET sam)
add_executable(${TEST_TARGET} sam.cpp)
target_link_libraries(${TEST_TARGET} PRIVATE ggml common)
```

--------------------------------

### Configure MUSA Architectures and Headers

Source: https://github.com/ggml-org/ggml/blob/master/src/ggml-musa/CMakeLists.txt

Sets default MUSA architectures if not defined and collects MUSA-specific header files.

```cmake
if (MUSAToolkit_FOUND)
    message(STATUS "MUSA Toolkit found")

    if (NOT DEFINED MUSA_ARCHITECTURES)
        set(MUSA_ARCHITECTURES "21;22;31")
    endif()
    message(STATUS "Using MUSA architectures: ${MUSA_ARCHITECTURES}")

    file(GLOB   GGML_HEADERS_MUSA "../ggml-cuda/*.cuh")
    list(APPEND GGML_HEADERS_MUSA "../../include/ggml-cuda.h")
    list(APPEND GGML_HEADERS_MUSA "../ggml-musa/mudnn.cuh")
```