### JavaScript Project Setup and Testing Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Install project dependencies using npm and run tests. ```bash npm install npm test ``` -------------------------------- ### Installation Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/10-python-api.md Instructions for installing the StringZilla library and its optional backends for CPU and GPU acceleration. ```bash pip install stringzilla pip install stringzillas-cpus pip install stringzillas-cuda ``` -------------------------------- ### Example of Stateful Hasher Usage Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Provides a concrete example of using the Hasher with specific string data, showing how to update and digest to get the final hash. ```javascript const hasher = new stringzilla.Hasher(42); hasher.update(Buffer.from('hello ')); hasher.update(Buffer.from('world')); const result = hasher.digest(); // Same as stringzilla.hash(Buffer.from('hello world'), 42) ``` -------------------------------- ### Example: Initialize CPU Cores Device Scope Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/09-parallel-api.md Example demonstrating how to initialize a device scope to use 4 CPU cores. ```c szs_device_scope_t scope = NULL; szs_device_scope_init_cpu_cores(4, &scope, NULL); // Use 4 cores ``` -------------------------------- ### Configure Installation Directories Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Sets up include directories for the build and installation paths. ```cmake include(GNUInstallDirs) set(STRINGZILLA_INCLUDE_BUILD_DIR "${PROJECT_SOURCE_DIR}/include/") set(STRINGZILLA_INCLUDE_INSTALL_DIR "${CMAKE_INSTALL_INCLUDEDIR}") ``` -------------------------------- ### Example: Levenshtein Distance Calculation with u32tape Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/09-parallel-api.md Demonstrates initializing the Levenshtein engine and computing distances between two sequences using the u32tape format. Includes setup for sequences, device scope, and output array. ```c #include int main() { szs_levenshtein_distances_t engine = NULL; szs_levenshtein_distances_init(0, 1, 1, 1, NULL, sz_cap_serial_k, &engine); // Setup tape with strings const char *data_a = "kittensitting"; // offsets: 0, 6 const char *data_b = "flowmlawn"; // offsets: 0, 5 sz_u32_t offsets_a[] = {0, 6, 13}; sz_u32_t offsets_b[] = {0, 5, 9}; sz_sequence_u32tape_t tape_a = {data_a, offsets_a, 2}; sz_sequence_u32tape_t tape_b = {data_b, offsets_b, 2}; szs_device_scope_t scope = NULL; szs_device_scope_init_default(&scope, NULL); sz_size_t distances[2]; szs_levenshtein_distances_u32tape(engine, scope, &tape_a, &tape_b, distances, sizeof(distances[0])); // distances[0] = 3 (kitten -> sitting) // distances[1] = 2 (flaw -> lawn) return 0; } ``` -------------------------------- ### Install StringZilla from Source Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/10-python-api.md Clone the repository and install StringZilla using pip. Supports installation with default options, development mode, and specific backends like CPU or CUDA parallelism. ```bash git clone https://github.com/ashvardanian/StringZilla.git cd StringZilla pip install . # Install with default options pip install -e . # Development mode # With specific backends pip install .[cpus] # With CPU parallelism pip install .[cuda] # With CUDA support ``` -------------------------------- ### Install StringZilla Headers and Source Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Installs StringZilla's include directory and C source files to their respective installation destinations if STRINGZILLA_INSTALL is enabled. ```cmake install(DIRECTORY ${STRINGZILLA_INCLUDE_BUILD_DIR} DESTINATION ${STRINGZILLA_INCLUDE_INSTALL_DIR}) install(DIRECTORY c/ DESTINATION ${CMAKE_INSTALL_DATADIR}/${PROJECT_NAME}/src) ``` -------------------------------- ### Install StringZilla Core and Backends Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/10-python-api.md Install the core StringZilla library for serial algorithms and optional parallel backends for CPU or CUDA. ```bash pip install stringzilla # Core library (serial algorithms) pip install stringzillas-cpus # Parallel CPU backends pip install stringzillas-cuda # Parallel CUDA/GPU backend ``` -------------------------------- ### Install and Check Capabilities (JavaScript) Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Install the Node.js package and check StringZilla's capabilities for CommonJS and ESM environments. ```bash npm install stringzilla node -p "require('stringzilla').default.capabilities" # for CommonJS node -e "import('stringzilla').then(m=>console.log(m.default.capabilities)).catch(console.error)" # for ESM ``` -------------------------------- ### Set Up Python Virtual Environment and Install Dependencies Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Create a Python virtual environment using 'uv', activate it, and install necessary build tools and StringZilla from source. This ensures a clean and isolated development environment. ```bash uv venv --python 3.12 # or your preferred Python version source .venv/bin/activate # to activate the virtual environment uv pip install setuptools wheel # to pull the latest build tools uv pip install -e . --force-reinstall # to build locally from source ``` -------------------------------- ### Install StringZilla Bare Library Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Installs the StringZilla bare library if the STRINGZILLA_INSTALL option is enabled and the target exists. Specifies installation types. ```cmake if (TARGET stringzilla_bare) install( TARGETS stringzilla_bare ARCHIVE BUNDLE FRAMEWORK LIBRARY OBJECTS PRIVATE_HEADER PUBLIC_HEADER RESOURCE RUNTIME ) endif () ``` -------------------------------- ### Install StringZilla JavaScript/Node.js with Prebuilt Binaries Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/14-compilation-config.md Install StringZilla for JavaScript/Node.js. Use 'npm view stringzilla dist.tarball' to list available platforms. Force a rebuild from source using '--build-from-source' if needed. ```bash # List available platforms npm view stringzilla dist.tarball # Force rebuild from source npm install stringzilla --build-from-source ``` -------------------------------- ### Get StringZilla Version and Capabilities Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/10-python-api.md Import the stringzilla library to check the installed version and the available hardware optimizations. ```python import stringzilla as sz print(sz.__version__) # Library version print(sz.__capabilities__) # Available hardware optimizations ``` -------------------------------- ### Install StringZillas CPUs Shared Library Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Installs StringZilla's CPUs shared library if it was built and STRINGZILLA_INSTALL is enabled. Specifies installation types. ```cmake # Install StringZillas shared libraries if they were built if (TARGET stringzillas_cpus_shared) install( TARGETS stringzillas_cpus_shared ARCHIVE BUNDLE FRAMEWORK LIBRARY OBJECTS PRIVATE_HEADER PUBLIC_HEADER RESOURCE RUNTIME ) endif () ``` -------------------------------- ### Build StringZilla from Source Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Instructions for cloning the repository, installing dependencies, building the project, running tests, and creating a distribution package using npm. ```bash # Clone repository git clone https://github.com/ashvardanian/StringZilla.git cd StringZilla # Install and build npm install npm run build # Run tests npm test # Create distribution package npm pack ``` -------------------------------- ### StringZilla Installation and Build Options Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Configures installation and build options for StringZilla, such as enabling tests, benchmarks, shared libraries, and sanitizers. Defaults are often tied to whether StringZilla is the main project. ```cmake option(STRINGZILLA_INSTALL "Install CMake targets" OFF) option(STRINGZILLA_BUILD_TEST "Compile a native unit test in C++" ${STRINGZILLA_IS_MAIN_PROJECT}) option(STRINGZILLA_BUILD_BENCHMARK "Compile a native benchmark in C++" ${STRINGZILLA_IS_MAIN_PROJECT}) option(STRINGZILLA_BUILD_SHARED "Compile a dynamic library" ${STRINGZILLA_IS_MAIN_PROJECT}) option(STRINGZILLAS_BUILD_SHARED "Compile dynamic parallel libraries" ${STRINGZILLA_IS_MAIN_PROJECT}) option(STRINGZILLA_BUILD_CUDA "Build CUDA-accelerated targets" ${STRINGZILLA_CAN_BUILD_CUDA}) option(STRINGZILLA_USE_SANITIZERS "Enable AddressSanitizer and UndefinedBehaviorSanitizer in Debug builds" ON) ``` -------------------------------- ### GoLang Quick Start: Build Shared Library Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Instructions for building the shared C library for Go bindings. Ensure the runtime can locate the library. ```bash cmake -B build_shared -D STRINGZILLA_BUILD_SHARED=1 -D CMAKE_BUILD_TYPE=Release cmake --build build_shared --target stringzilla_shared --config Release export LD_LIBRARY_PATH="$PWD/build_shared:$LD_LIBRARY_PATH" ``` -------------------------------- ### Install StringZilla Shared Library Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Installs the StringZilla shared library if the STRINGZILLA_INSTALL option is enabled and the target exists. Specifies installation types. ```cmake if (STRINGZILLA_INSTALL) if (TARGET stringzilla_header) install( TARGETS stringzilla_shared ARCHIVE BUNDLE FRAMEWORK LIBRARY OBJECTS PRIVATE_HEADER PUBLIC_HEADER RESOURCE RUNTIME ) endif () ``` -------------------------------- ### Example: Array of C Strings Sequence Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/07-c-api-sort.md Implements `get_start` and `get_length` callbacks for an array of C strings to be used with `sz_sequence_t`. ```c const char *strings[] = {"zebra", "apple", "mango"}; sz_cptr_t get_start(void *handle, sz_size_t i) { return ((const char **)handle)[i]; } sz_size_t get_length(void *handle, sz_size_t i) { return strlen(((const char **)handle)[i]); } sz_sequence_t seq = { .start = (void *)strings, .count = 3, .get_start = get_start, .get_length = get_length }; ``` -------------------------------- ### StringZilla Benchmarks Setup Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Instructions for running StringZilla benchmarks. Benchmarks are available in the './scripts' directory. Compilation uses GCC 12 and glibc v2.35. ```bash CONTRIBUTING.md ``` -------------------------------- ### Install Python Package Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/00-index.md Installs the StringZilla Python package, with optional packages for CPU and GPU parallelism. ```bash pip install stringzilla pip install stringzillas-cpus # With parallelism pip install stringzillas-cuda # With GPU ``` -------------------------------- ### Version and Capabilities Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/10-python-api.md How to check the installed StringZilla library version and its available hardware capabilities. ```python import stringzilla as sz print(sz.__version__) print(sz.__capabilities__) ``` -------------------------------- ### Complete Rust Program Example Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/11-rust-api.md A comprehensive example program showcasing various stringzilla functionalities including basic search, hashing, comparison, UTF-8 case folding, and parallel Levenshtein distance computation. ```rust use stringzilla as sz; fn main() -> Result<(), Box> { // Basic search let text = "The quick brown fox jumps"; let pos = sz::find(text, "brown")?; println!("Found 'brown' at position {}", pos); // Hashing let hash = sz::hash(b"quick", 0)?; println!("Hash: {}", hash); // Comparison let equal = sz::equal("fox", "fox")?; assert!(equal); // UTF-8 operations let folded = sz::utf8_case_fold("Straße")?; println!("Folded: {:?}", folded); #[cfg(feature = "cpus")] { // Parallel operations use stringzilla::szs::LevenshteinDistances; let strings_a = vec!["kitten"]; let strings_b = vec!["sitting"]; let distances = LevenshteinDistances { match_cost: 0, mismatch_cost: 1, gap_open_cost: 1, gap_extend_cost: 1, }; let results = distances.compute(&strings_a, &strings_b)?; println!("Distance: {}", results[0]); } Ok(()) } ``` -------------------------------- ### Install PyTest and Run Tests Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Install PyTest and related packages for running StringZilla's Python tests. Includes commands to execute tests with default or custom settings, and to run a specific hash function test. ```bash uv pip install pytest pytest-repeat numpy pyarrow # for repeated fuzzy tests uv run --no-project python -m pytest scripts/test_stringzilla.py # to run with default settings uv run --no-project python -m pytest scripts/test_stringzilla.py -s -x -p no:warnings # to pass custom settings uv run --no-project python -c 'from stringzilla import hash as sz_hash; print(sz_hash("abc", 100))' ``` -------------------------------- ### Complete JavaScript Program Example Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md A comprehensive example demonstrating various stringzilla functionalities including basic search, hashing, streaming hash computation, SHA-256 digest, pattern counting, character set search, and case-insensitive UTF-8 search. ```javascript import stringzilla from 'stringzilla'; // Basic search const text = Buffer.from('The quick brown fox jumps'); const needle = Buffer.from('brown'); const pos = stringzilla.find(text, needle); console.log(`Found at position: ${pos}`); // 10n // Hashing const hash = stringzilla.hash(text, 42); console.log(`Hash: ${hash}`); // Streaming hash const hasher = new stringzilla.Hasher(0); hasher.update(Buffer.from('hello ')); hasher.update(Buffer.from('world')); console.log(`Streamed hash: ${hasher.digest()}`); // SHA-256 const digest = stringzilla.sha256(text); console.log(`SHA256: ${digest.toString('hex')}`); // Counting patterns const patterns = Buffer.from('the'); const count = stringzilla.count(text, patterns); console.log(`'the' appears ${count} time(s)`); // Character set search const vowels = Buffer.from('aeiouAEIOU'); const firstVowel = stringzilla.findByteFrom(text, vowels); console.log(`First vowel at: ${firstVowel}`); // UTF-8 case-insensitive find (if supported) if (stringzilla.utf8_case_insensitive_find) { const caseless = stringzilla.utf8_case_insensitive_find( Buffer.from('Hello World'), Buffer.from('WORLD') ); console.log(`Found (case-insensitive): ${caseless}`); } ``` -------------------------------- ### Install Node.js Package Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/00-index.md Installs the StringZilla package for use in Node.js projects via npm. ```bash npm install stringzilla ``` -------------------------------- ### Install Optional Python Dependencies for Testing Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Install optional dependencies required for benchmarking and other scripts related to StringZilla's testing suite. This command reads requirements from a file. ```bash uv pip install -r scripts/requirements.txt ``` -------------------------------- ### StringZilla C API Dispatch Examples Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Demonstrates various C functions for string searching with different SIMD backends, from auto-dispatch to specific Intel and ARM architectures. ```c sz_find(text, length, pattern, 3); // Auto-dispatch sz_find_westmere(text, length, pattern, 3); // Intel Westmere+ SSE4.2 sz_find_haswell(text, length, pattern, 3); // Intel Haswell+ AVX2 sz_find_skylake(text, length, pattern, 3); // Intel Skylake+ AVX-512 sz_find_neon(text, length, pattern, 3); // Arm NEON 128-bit sz_find_sve(text, length, pattern, 3); // Arm SVE 128/256/512/1024/2048-bit ``` -------------------------------- ### Install StringZilla Python Package Source: https://github.com/ashvardanian/stringzilla/blob/main/cli/README.md Install the StringZilla Python package using pip. This is the primary method for accessing the CLI utilities. ```bash pip install stringzilla ``` -------------------------------- ### String View Usage Example (C) Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/13-types-reference.md Demonstrates the creation and usage of a `sz_string_view_t` to represent a string segment and calculate relative offsets. ```c sz_string_view_t view = {data_ptr, data_length}; sz_size_t offset = ptr - view.start; // Relative position ``` -------------------------------- ### Install Static Analysis Tools Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Installs CppCheck and Clang-Tidy for static code analysis. These tools can be noisy and are best suited for local development. ```bash sudo apt install cppcheck clang-tidy-11 ``` -------------------------------- ### Install CMake and Compilers on Ubuntu Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Installs necessary build tools including CMake and specific GCC versions for compiling C++ code on Ubuntu. ```bash sudo apt-get update sudo apt-get install build-essential sudo apt-get install cmake # Consider pulling a newer version from PyPI sudo apt-get install g++-12 gcc-12 # You may already have a newer version on Ubuntu 24 sudo apt install libstdc++6-12-dbg # STL debugging symbols for GCC 12 ``` -------------------------------- ### Complete C++ Program Example Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Demonstrates common string operations in C++ using the stringzilla library. Includes string creation, extension checking, partitioning, finding characters, and case-insensitive searching. ```cpp #include #include namespace sz = ashvardanian::stringzilla; int main() { // Create strings sz::string filename = "report_2024.txt"; // Check extension if (filename.ends_with(".txt")) { std::cout << "Text file detected\n"; } // Extract name (split around dot) auto [name, _, ext] = filename.rpartition('.'); std::cout << "Name: " << name << ", Ext: " << ext << "\n"; // Find all vowels sz::byteset vowels("aeiouAEIOU"); for (auto match : name.find_all(vowels)) { std::cout << "Vowel: " << match << "\n"; } // Case-insensitive substring sz::utf8_case_insensitive_needle pattern("REPORT"); auto [offset, len] = filename.utf8_case_insensitive_find(pattern); std::cout << "Found at byte " << offset << "\n"; return 0; } ``` -------------------------------- ### Build Source Distributions with UV Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Use `uv` to install build tools and then build source distributions for different StringZilla variants by setting the `SZ_TARGET` environment variable. ```bash uv pip install build uv build --sdist --out-dir dist # defaults to `stringzilla` SZ_TARGET=stringzilla uv run --no-project python build_backend.py build-sdists SZ_TARGET=stringzillas-cpus uv run --no-project python build_backend.py build-sdists SZ_TARGET=stringzillas-cuda uv run --no-project python build_backend.py build-sdists ``` -------------------------------- ### Profile StringZilla Benchmarks Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Build StringZilla with RelWithDebInfo configuration to include debugging symbols for profiling. This example targets the `stringzilla_bench_token_cpp20` benchmark and demonstrates how to record and report performance data using `perf`. ```bash cmake -D STRINGZILLA_BUILD_BENCHMARK=1 \ -D STRINGZILLA_BUILD_TEST=1 \ -D STRINGZILLA_BUILD_SHARED=1 \ -D CMAKE_BUILD_TYPE=RelWithDebInfo \ -B build_profile cmake --build build_profile --config Release --target stringzilla_bench_token_cpp20 # Check that the debugging symbols are there with your favorite tool readelf --sections build_profile/stringzilla_bench_token_cpp20 | grep debug objdump -h build_profile/stringzilla_bench_token_cpp20 | grep debug # Profile sudo perf record -g build_profile/stringzilla_bench_token_cpp20 ./leipzig1M.txt sudo perf report ``` -------------------------------- ### Check StringZilla Version and Capabilities Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Verify the installed StringZilla version and its hardware capabilities using these Python commands. ```python python -c "import stringzilla; print(stringzilla.__version__)" ``` ```python python -c "import stringzillas; print(stringzillas.__version__)" ``` ```python python -c "import stringzilla; print(stringzilla.__capabilities__)" # for serial algorithms ``` ```python python -c "import stringzillas; print(stringzillas.__capabilities__)" # for parallel algorithms ``` -------------------------------- ### Python API Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/MANIFEST.txt Covers the Python API, including installation, the `Str` class, file operations, and parallel processing with `DeviceScope`. ```APIDOC ## Python API ### Description Documentation for the Python API, providing string manipulation and parallel processing capabilities. ### Installation - `stringzilla` - `stringzillas-cpus` - `stringzillas-cuda` ### Core Components - `Str` class - `File` for memory mapping ### Operations - Search - Splitting - Hashing - UTF-8 operations - Parallel operations with `DeviceScope` ### Integration - Integration with PyArrow for zero-copy buffers ``` -------------------------------- ### Build and Run C++ Tests on Windows with MinGW Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Installs MinGW build tools and then configures and builds the StringZilla C++ test suite using CMake with MinGW Makefiles. ```bash pacman -S --needed --noconfirm mingw-w64-x86_64-gcc mingw-w64-x86_64-cmake make cmake -G "MinGW Makefiles" -B build_release -D STRINGZILLA_BUILD_TEST=1 -D CMAKE_BUILD_TYPE=Release cmake --build build_release --config Release ./build_release/stringzilla_test_cpp20.exe ``` -------------------------------- ### Build and Test in Intel Clear Linux Docker Container Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Configure a build environment within a Clear Linux Docker container, optimized for Intel hardware. Installs development tools and builds StringZilla with Debug configuration for testing. ```bash sudo docker run -it --rm -v "$(pwd)":/workspace/StringZilla clearlinux:latest /bin/bash cd /workspace/StringZilla swupd update swupd bundle-add c-basic dev-utils cmake -D STRINGZILLA_BUILD_TEST=1 -D CMAKE_BUILD_TYPE=Debug -B build_debug cmake --build build_debug --config Debug build_debug/stringzilla_test_cpp20 ``` -------------------------------- ### StringZilla C API Usage Example Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Demonstrates common operations using the StringZilla C API, including initialization, heap allocation, appending, erasing, unpacking string information, and freeing memory. Ensure proper memory management with the provided allocator. ```c sz_memory_allocator_t allocator; sz_string_t string; // Init and make sure we are on stack sz_string_init(&string); sz_string_is_on_stack(&string); // == sz_true_k // Optionally pre-allocate space on the heap for future insertions. sz_string_grow(&string, 100, &allocator); // == sz_true_k // Append, erase, insert into the string. sz_string_expand(&string, 0, "_Hello_", 7, &allocator); // == sz_true_k sz_string_expand(&string, SZ_SIZE_MAX, "world", 5, &allocator); // == sz_true_k sz_string_erase(&string, 0, 1); // Unpacking & introspection. sz_ptr_t string_start; sz_size_t string_length; sz_size_t string_space; sz_bool_t string_is_external; sz_string_unpack(string, &string_start, &string_length, &string_space, &string_is_external); sz_equal(string_start, "Hello_world", 11); // == sz_true_k // Reclaim some memory. sz_string_shrink_to_fit(&string, &allocator); // == sz_true_k sz_string_free(&string, &allocator); ``` -------------------------------- ### StringZilla Python Backend Reset Examples Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Illustrates how to explicitly set or reset the SIMD backend for StringZilla operations in Python, including forcing specific architectures or resetting to auto-dispatch. ```python import stringzilla as sz sz.reset_capabilities(('serial',)) # Force SWAR backend ``` ```python sz.reset_capabilities(('haswell',)) # Force AVX2 backend ``` ```python sz.reset_capabilities(('neon',)) # Force NEON backend ``` ```python sz.reset_capabilities(sz.__capabilities__) # Reset to auto-dispatch ``` -------------------------------- ### Check StringZilla Python Capabilities Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Verify the installed version and capabilities of the StringZilla Python package. This command executes a Python script to print the capabilities. ```bash uv run --no-project python -c "import stringzilla as sz; print(sz.__capabilities__)" ``` -------------------------------- ### Find Nth UTF-8 Codepoint Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/06-c-api-utf8.md Use `sz_utf8_find_nth` to get a byte-level pointer to the start of the Nth UTF-8 codepoint in a string. The position is 0-based and counts codepoints, not bytes. Returns NULL if the position is out of bounds. ```c const char *text = "café"; // 4 codepoints: c, a, f, é sz_cptr_t third = sz_utf8_find_nth(text, 4, 2); // Points to 'f' ``` -------------------------------- ### Integrating with ndarray in Rust Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/11-rust-api.md Demonstrates how to use Rust's `ndarray` crate for numerical computing and convert its types for use with the stringzilla API. This example shows initializing a 2D array. ```rust // Use with ndarray types use ndarray::Array2; let matrix: Array2 = Array2::zeros((256, 256)); // Convert to &[[i8; 256]; 256] for use with API ``` -------------------------------- ### Build and Test in Alpine Docker Container Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Set up a build environment within an Alpine Linux Docker container. Installs necessary build tools (make, cmake, g++) and builds StringZilla with Debug configuration for testing. ```bash sudo docker run -it --rm -v "$(pwd)":/workspace/StringZilla alpine:latest /bin/ash cd /workspace/StringZilla apk add --update make cmake g++ gcc cmake -D STRINGZILLA_BUILD_TEST=1 -D CMAKE_BUILD_TYPE=Debug -B build_debug cmake --build build_debug --config Debug build_debug/stringzilla_test_cpp20 ``` -------------------------------- ### Install StringZillas CUDA Shared Library Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Installs StringZilla's CUDA shared library if it was built and STRINGZILLA_INSTALL is enabled. Specifies installation types. ```cmake if (TARGET stringzillas_cuda_shared) install( TARGETS stringzillas_cuda_shared ARCHIVE BUNDLE FRAMEWORK LIBRARY OBJECTS PRIVATE_HEADER PUBLIC_HEADER RESOURCE RUNTIME ) endif () ``` -------------------------------- ### Initialize and Use Fingerprinting Engine in C Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Demonstrates initializing a device scope, fingerprinting engine, and processing text data using Min-Hashes and Count-Min-Sketches. Ensure proper allocation and deallocation of resources. ```c #include szs_device_scope_t device = NULL; szs_device_scope_init_default(&device); szs_fingerprints_t engine = NULL; sz_size_t const dims = 1024; sz_size_t const window_widths[] = {4, 6, 8, 10}; szs_fingerprints_init(dims, /*alphabet*/ 256, window_widths, 4, /*alloc*/ NULL, /*caps*/ sz_cap_serial_k, &engine); sz_sequence_u32tape_t texts = {data, offsets, count}; sz_u32_t *min_hashes = (sz_u32_t*)szs_unified_alloc(count * dims * sizeof(*min_hashes)); sz_u32_t *min_counts = (sz_u32_t*)szs_unified_alloc(count * dims * sizeof(*min_counts)); szs_fingerprints_u32tape(engine, device, &texts, min_hashes, dims * sizeof(*min_hashes), // support strided matrices min_counts, dims * sizeof(*min_counts)); // for both output arguments szs_fingerprints_free(engine); szs_device_scope_free(device); ``` -------------------------------- ### Benchmark GoLang Module Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Navigate to the GoLang module directory and run benchmarks. Set CGO flags and specify the input file. ```bash cd golang CGO_CFLAGS="-I$(pwd)/../include" CGO_LDFLAGS="-L$(pwd)/../build_golang -lstringzilla_shared" LD_LIBRARY_PATH="$(pwd)/../build_golang:$LD_LIBRARY_PATH" go run ../scripts/bench.go --input ../leipzig1M.txt ``` -------------------------------- ### Initialize string_view Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Demonstrates how to initialize a `string_view` from a C-style string literal or a buffer and length. This view is read-only and does not own the string data. ```cpp namespace sz = ashvardanian::stringzilla; sz::string_view text = "hello"; // or sz::string_view text(buffer, length); ``` -------------------------------- ### Initialize and Use Stateful Hasher Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Demonstrates how to initialize a Hasher with a seed, update it with chunks of data, and then retrieve the final hash value or its hex representation. ```javascript const hasher = new stringzilla.Hasher(seed); hasher.update(chunk1); hasher.update(chunk2); const hash = hasher.digest(); // bigint const hex = hasher.hexdigest(); // string: hex representation ``` -------------------------------- ### Build StringZilla with CPU and CUDA Backends for Python Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Install StringZilla with specific backends (CPUs or CUDA) for Python. This involves setting the SZ_TARGET environment variable before running pip install. ```bash uv pip install setuptools wheel numpy SZ_TARGET=stringzillas-cpus uv pip install -e . --force-reinstall --no-build-isolation SZ_TARGET=stringzillas-cuda uv pip install -e . --force-reinstall --no-build-isolation ``` -------------------------------- ### Initialize and Use Stateful Sha256 Hasher Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Shows how to create a Sha256 hasher instance, update it with data chunks, and obtain the resulting digest as a Buffer or a hex string. ```javascript const sha = new stringzilla.Sha256(); sha.update(chunk1); sha.update(chunk2); const digest = sha.digest(); // Buffer: 32 bytes const hex = sha.hexdigest(); // string: 64 hex chars ``` -------------------------------- ### Implement and Use Generic Sequence Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/13-types-reference.md Demonstrates implementing get_start and get_length callbacks for a C-style array of strings and initializing an sz_sequence_t. This allows StringZilla to process the array as a sequence. ```c const char *strings[] = {"foo", "bar", "baz"}; sz_cptr_t get_start(void *handle, sz_size_t i) { return ((const char **)handle)[i]; } sz_size_t get_length(void *handle, sz_size_t i) { return strlen(((const char **)handle)[i]); } sz_sequence_t seq = { .start = (void *)strings, .count = 3, .get_start = get_start, .get_length = get_length }; ``` -------------------------------- ### Initialize Default Device Scope Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/09-parallel-api.md Initializes device scope with system defaults, utilizing all available hardware. Returns SZ_OK on success. ```c SZ_DYNAMIC sz_status_t szs_device_scope_init_default(szs_device_scope_t *scope, char const **error_message); ``` -------------------------------- ### StringZilla Case-Insensitive UTF-8 Search Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Provides examples of StringZilla's case-insensitive UTF-8 search functionality. It returns the byte offset of matches and correctly handles character expansions. Includes examples for finding all matches using an iterator and handling overlapping matches. ```python import stringzilla as sz sz.utf8_case_insensitive_find('Der große Hund', 'GROSSE') # 4 — finds "große" at codepoint 4 sz.utf8_case_insensitive_find('Straße', 'STRASSE') # 0 — ß matches "SS" sz.utf8_case_insensitive_find('efficient', 'EFFICIENT') # 0 — ffi ligature matches "FFI" # Iterator for finding ALL matches haystack = 'Straße STRASSE strasse' for match in sz.utf8_case_insensitive_find_iter(haystack, 'strasse'): print(match, match.offset_within(haystack)) # Yields: 'Straße', 'STRASSE', 'strasse' # With overlapping matches list(sz.utf8_case_insensitive_find_iter('aaaa', 'aa')) # ['aa', 'aa'] — 2 non-overlapping list(sz.utf8_case_insensitive_find_iter('aaaa', 'aa', include_overlapping=True)) # 3 matches ``` -------------------------------- ### find Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Finds the starting position of a needle buffer within a text buffer. ```APIDOC ## find ### Description Finds the starting position of a needle within a text. ### Method Not specified (assumed to be a function call within JavaScript) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **text** (Buffer) - The buffer to search within. - **needle** (Buffer) - The buffer to search for. ### Response #### Success Response - **pos** (BigInt) - The starting position of the needle in the text. Returns a BigInt. ### Request Example ```javascript const text = Buffer.from('The quick brown fox jumps'); const needle = Buffer.from('brown'); const pos = stringzilla.find(text, needle); console.log(`Found at position: ${pos}`); // 10n ``` ``` -------------------------------- ### Run GoLang Module Tests Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Navigate to the GoLang module directory and run tests. Set CGO flags to link the shared library. ```bash cd golang CGO_CFLAGS="-I$(pwd)/../include" CGO_LDFLAGS="-L$(pwd)/../build_golang -lstringzilla_shared" LD_LIBRARY_PATH="$(pwd)/../build_golang:$LD_LIBRARY_PATH" go test ``` -------------------------------- ### Example Error Cost Values Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/13-types-reference.md Illustrates the usage of `sz_error_cost_t` with common values for match, mismatch, and gap costs. ```c sz_error_cost_t match_cost = 0; sz_error_cost_t mismatch_cost = 1; sz_error_cost_t gap_open = 2; sz_error_cost_t gap_extend = 1; ``` -------------------------------- ### Define Header-Only Library Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Defines an interface library for the header-only version of StringZilla. It specifies include directories for build and installation. ```cmake add_library(stringzilla_header INTERFACE) add_library(${PROJECT_NAME}::stringzilla_header ALIAS stringzilla_header) target_include_directories( stringzilla_header INTERFACE $ $ ) ``` -------------------------------- ### Create and Use byteset Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Demonstrates the creation and basic usage of a `sz::byteset` for efficient bitmap representation of byte values. Use this for quick lookups of byte presence. ```cpp sz::byteset set; set['a'] = true; set[0] = false; bool contains_a = set['a']; // bool ``` -------------------------------- ### Swift Project Build and Test Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Build and test a Swift project using `swift build` and `swift test`. ```bash swift build && swift test ``` -------------------------------- ### Boolean Type Usage Example (C) Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/13-types-reference.md Shows how to assign and check the custom boolean type `sz_bool_t` for conditional logic. ```c sz_bool_t found = (result != NULL) ? sz_true_k : sz_false_k; if (found == sz_true_k) { ... } ``` -------------------------------- ### JavaScript/Node.js API Usage Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/README.md Example of using the stringzilla Node.js native addon for string searching. Import the stringzilla module. ```javascript import stringzilla from 'stringzilla'; const pos = stringzilla.find(haystack, needle); ``` -------------------------------- ### Rust API Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/MANIFEST.txt Details the Rust API, including installation, core modules for single-string operations, and parallel modules for advanced algorithms. ```APIDOC ## Rust API ### Description Documentation for the Rust API, covering single-string and parallel operations. ### Installation - Feature flags (cpus, cuda, rocm) ### Modules - `stringzilla::sz` (Core module for single-string operations) - `stringzilla::szs` (Parallel module for distance/alignment/fingerprints) ### Considerations - Error handling - Performance considerations ### Examples - Complete example programs ``` -------------------------------- ### Initialize Git Submodules Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Run this command before building for the first time to pull in necessary submodules like fork_union for full functionality testing. ```sh git submodule update --init --recursive ``` -------------------------------- ### Unsafe String Construction Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Provides examples of using unchecked operations for performance-critical code, bypassing bounds checks during string modifications. ```cpp using sz::string::unchecked; str.push_back('x', unchecked); // no bounds check str.insert(pos, "text", unchecked); // no bounds check ``` -------------------------------- ### Create byteset from String or Preset Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Shows how to initialize a `sz::byteset` either from a string containing characters or from a predefined set like `sz::whitespaces_set()`. ```cpp sz::byteset set(" \t\n\r\v\f"); // from string sz::byteset set(sz::whitespaces_set()); // from preset ``` -------------------------------- ### Checking StringZilla Sort Status Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/07-c-api-sort.md Example of how to check the return status of `sz_sequence_argsort` and handle errors. Ensure you check for `sz_ok_k` before proceeding. ```c sz_status_t status = sz_sequence_argsort(&sequences, NULL, order); if (status != sz_ok_k) { fprintf(stderr, "Sort failed with status: %d\n", status); return 1; } ``` -------------------------------- ### Initialize Fingerprinting Engine Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/09-parallel-api.md Creates a fingerprinting engine with specified window widths. Requires defining the number of hash functions, alphabet size, window sizes, an optional memory allocator, and hardware capabilities. ```c SZ_DYNAMIC sz_status_t szs_fingerprints_init( sz_size_t dimensions, sz_size_t alphabet_size, sz_size_t const *window_widths, sz_size_t window_widths_count, sz_memory_allocator_t *alloc, sz_capability_t capabilities, szs_fingerprints_t *engine ); ``` -------------------------------- ### Initialize Banned Characters for Alignment Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Initializes a substitution matrix with penalties for banned characters. This is a setup step before performing sequence alignment. ```python subs_reconstructed.fill(127) for packed_row, packed_row_aminoacid in enumerate(aligner.substitution_matrix.alphabet): for packed_column, packed_column_aminoacid in enumerate(aligner.substitution_matrix.alphabet): reconstructed_row = ord(packed_row_aminoacid) reconstructed_column = ord(packed_column_aminoacid) subs_reconstructed[reconstructed_row, reconstructed_column] = subs_packed[packed_row, packed_column] ``` -------------------------------- ### Example of Stateful Sha256 Hasher Usage Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/12-javascript-api.md Illustrates the practical application of the Sha256 hasher by updating it with string data and retrieving the computed digest. ```javascript const sha = new stringzilla.Sha256(); sha.update(Buffer.from('hello ')); sha.update(Buffer.from('world')); const digest = sha.digest(); // Same as stringzilla.sha256(Buffer.from('hello world')) ``` -------------------------------- ### string_view Slicing and Partitioning Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Shows how to extract substrings using `substr`, `front`, `back`, and Python-style slicing. Also demonstrates partitioning the view around a delimiter. ```cpp text.substr(pos, len); // substring (returns new string_view) text.front(n); // first n characters text.back(n); // last n characters text.sub(start, end); // Python-style slicing [start:end] auto [before, match, after] = text.partition(':'); // split around first match auto [before, match, after] = text.rpartition(':'); // split around last match ``` -------------------------------- ### Check Runtime Capabilities with Python Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/14-compilation-config.md Import the stringzilla library in Python and print the __capabilities__ attribute to see which backends are available at runtime. This output reflects the dynamically detected CPU features. ```python import stringzilla as sz # See all available backends print(sz.__capabilities__) # Output: ('serial', 'westmere', 'haswell', 'skylake') ``` -------------------------------- ### Device Scope Examples Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/11-rust-api.md Specify the hardware for parallel operations. Defaults to all available hardware, but can be set to specific CPU cores or a GPU device. ```rust use stringzilla::szs::DeviceScope; // Default (all available hardware) let device = DeviceScope::default()?; // Specific CPU cores let device = DeviceScope::cpu_cores(4)?; // GPU device let device = DeviceScope::gpu_device(0)?; ``` -------------------------------- ### C++ STL Wrapper Usage Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/README.md Example of using the C++ STL-compatible wrapper for string searching and splitting. Requires the stringzilla C++ API. ```cpp namespace sz = ashvardanian::stringzilla; auto pos = text.find(needle); for (auto word : text.split(' ')) { ... } ``` -------------------------------- ### String Capacity and Sizing Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Shows how to query and manage the capacity of a sz::string, including checking for SSO, reducing allocation, and pre-allocating space. ```cpp str.size(); // current length str.capacity(); // allocated capacity str.is_on_stack(); // bool: SSO active str.shrink_to_fit(); // reduce allocation str.grow(new_capacity); // pre-allocate ``` -------------------------------- ### string_view Comparison and Hashing Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Illustrates how to compare `string_view` objects for ordering and equality, and how to obtain a hash value for use in hash-based containers. ```cpp text.compare(other); // -1/0/1 ordering text == other; // bool equality text < other; // bool comparison std::hash{}(text); // 64-bit hash ``` -------------------------------- ### BioPython to StringZilla Conversion Example Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Demonstrates converting a BioPython `PairwiseAligner` with a substitution matrix to a format compatible with StringZilla's Needleman-Wunsch engine. ```python import numpy as np from Bio import Align from Bio.Align import substitution_matrices aligner = Align.PairwiseAligner() aligner.substitution_matrix = substitution_matrices.load("BLOSUM62") aligner.open_gap_score = 1 aligner.extend_gap_score = 1 # Convert the matrix to NumPy subs_packed = np.array(aligner.substitution_matrix).astype(np.int8) subs_reconstructed = np.zeros((256, 256), dtype=np.int8) ``` -------------------------------- ### Get Developer User Name Source: https://github.com/ashvardanian/stringzilla/blob/main/CMakeLists.txt Retrieves the current user's name from the environment variable USER and stores it. This can be useful for logging or configuration. ```cmake set(DEV_USER_NAME $ENV{USER}) ``` -------------------------------- ### Swift Build and Test on Linux via Docker Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Build and test Swift projects on Linux using the official Swift Docker image, ensuring static Swift standard library linkage. ```bash sudo docker run --rm -v "$PWD:/workspace" -w /workspace swift:6.0 /bin/bash -cl "swift build -c release --static-swift-stdlib && swift test -c release" ``` -------------------------------- ### string_view Basic Operations Source: https://github.com/ashvardanian/stringzilla/blob/main/_autodocs/08-cpp-api.md Illustrates fundamental operations on a `string_view`, such as getting its size, data pointer, accessing individual characters, and using iterators. ```cpp text.size(); // size_t: byte length text.length(); // same as size() text.data(); // const char*: underlying pointer text[0]; // char: first byte text.front(); // char: first character text.back(); // char: last character text.begin(), text.end(); // iterators text.rbegin(), text.rend(); // reverse iterators ``` -------------------------------- ### Configure CMake for Benchmarking Source: https://github.com/ashvardanian/stringzilla/blob/main/CONTRIBUTING.md Configures the build system for StringZilla benchmarks. This command prepares the build directory for compiling benchmark targets. ```bash cmake -D STRINGZILLA_BUILD_BENCHMARK=1 -B build_release ``` -------------------------------- ### SHA-256 Checksums (JavaScript) Source: https://github.com/ashvardanian/stringzilla/blob/main/README.md Compute SHA-256 cryptographic checksums using one-shot or incremental methods. Get results as a Buffer or hex string. ```javascript import sz from 'stringzilla'; // One-shot SHA-256 const digest = sz.sha256(Buffer.from('Hello, world!')); // returns Buffer (32 bytes) // Incremental SHA-256 const hasher = new sz.Sha256(); hasher.update(Buffer.from('Hello, ')); hasher.update(Buffer.from('world!')); const digestBuffer = hasher.digest(); // returns Buffer (32 bytes) const digestHex = hasher.hexdigest(); // returns string (64 hex chars) ```