### Install Pyppmd using pip Source: https://github.com/miurahr/pyppmd/blob/main/docs/getting_started.md Use this command to install the pyppmd library from PyPI. It automatically selects the correct wheel based on your Python environment (CPython or PyPy). ```console pip install pyppmd ``` -------------------------------- ### Create Virtual Environment and Install Dependencies Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Defines a custom command to create a Python virtual environment and install dependencies from `requirements.txt`. A `venv.stamp` file is created upon successful completion to track the state. ```cmake add_custom_command(OUTPUT venv.stamp COMMAND ${Python_EXECUTABLE} -m venv ${VENV_PATH} COMMAND ${PIP_COMMAND} install -r ${CMAKE_BINARY_DIR}/requirements.txt COMMAND ${CMAKE_COMMAND} -E touch venv.stamp) ``` -------------------------------- ### Set Minimum CMake Version and Project Name Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Specifies the minimum required CMake version and names the project. This is a standard starting point for any CMake project. ```cmake cmake_minimum_required(VERSION 3.19) project(pyppmd C CXX) ``` -------------------------------- ### Set Pip Command for Virtual Environment Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Determines the correct `pip` command based on the operating system to install packages into the virtual environment. This ensures the correct pip executable is used. ```cmake if (WIN32) set(PIP_COMMAND ${VENV_PATH}/Scripts/pip.exe) else() set(PIP_COMMAND ${VENV_PATH}/bin/pip) endif() ``` -------------------------------- ### Determine Python Extension Suffix Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Generates and executes a Python script to determine the correct extension suffix for the current Python installation. This ensures compatibility across different platforms and Python versions. ```cmake file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/get_ext.py "import sysconfig\nprint(sysconfig.get_config_var('EXT_SUFFIX'))\n") execute_process( COMMAND ${Python_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/get_ext.py OUTPUT_VARIABLE PY_EXT_EXT OUTPUT_STRIP_TRAILING_WHITESPACE) ``` -------------------------------- ### Build PyPPMd Library Source: https://github.com/miurahr/pyppmd/blob/main/docs/contribution.md Compile the C files into a static library. This target is primarily for compilation convenience. ```bash cd cmake-build make pyppmd ``` -------------------------------- ### Define Requirements for Virtual Environment Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Writes a `requirements.txt` file to the build directory, listing all necessary Python packages for development and testing. This includes testing frameworks and CFFI. ```cmake file( WRITE ${CMAKE_CURRENT_BINARY_DIR}/requirements.txt "\ncoverage[toml]>=5.2\nhypothesis\npytest>=6.0\npytest-benchmark\npytest-cov\npytest-timeout\ncffi\npy-cpuinfo\ntox\n") ``` -------------------------------- ### Define Source Path and Site-Packages Directories Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Sets the source directory path and defines variables for the site-packages directories within the virtual environment. These paths are adjusted based on the Python implementation (CPython vs. PyPy) and version. ```cmake set(SRC_PATH "${CMAKE_SOURCE_DIR}/src") if ("${Python_INTERPRETER_ID}" STREQUAL "PyPy") set(VPKG_PATH_A "${VENV_PATH}/lib/pypy${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/site-packages/") else() set(VPKG_PATH_A "${VENV_PATH}/lib/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/site-packages/") endif() set(VPKG_PATH_B "${VENV_PATH}/Lib/site-packages/") ``` -------------------------------- ### Custom Target for Building and Copying Extension Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Defines a custom target `build_ext` that depends on the `_ppmd` library target. It copies the built extension to the source directory if it's different, ensuring the inplace build is up-to-date. ```cmake add_custom_target( build_ext BYPRODUCTS ${PY_EXT_INPLACE} COMMAND ${CMAKE_COMMAND} -E copy_if_different $ ${CMAKE_SOURCE_DIR}/src/${PY_EXT_DIR} DEPENDS _ppmd venv.stamp ) ``` -------------------------------- ### PPMD8 Restoration Method Constants and Usage Source: https://context7.com/miurahr/pyppmd/llms.txt Demonstrates the usage of PPMD8_RESTORE_METHOD_RESTART and PPMD8_RESTORE_METHOD_CUT_OFF constants, which control the model restoration strategy when memory is exhausted. These constants must be consistent between the encoder and decoder. ```python import pyppmd print(pyppmd.PPMD8_RESTORE_METHOD_RESTART) # 0 print(pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF) # 1 # Choosing restoration method affects compression ratio vs. stability data = b"Repetitive text " * 5000 for method, name in [ (pyppmd.PPMD8_RESTORE_METHOD_RESTART, "RESTART"), (pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF, "CUT_OFF"), ]: enc = pyppmd.Ppmd8Encoder(6, 1 << 20, restore_method=method) # intentionally small mem compressed = enc.encode(data) + enc.flush() dec = pyppmd.Ppmd8Decoder(6, 1 << 20, restore_method=method) out = dec.decode(compressed, -1) + dec.decode(b"", -1) print(f"{name}: compressed={len(compressed)} bytes, roundtrip_ok={out == data}") ``` -------------------------------- ### Configure Python Version and Virtual Environment Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Sets the target Python version and configures the virtual environment path. It also defines options for finding Python implementations. ```cmake set(PY_VERSION 3.12) set(Python_FIND_IMPLEMENTATIONS CPython) set(VENV_PATH "${CMAKE_BINARY_DIR}/venv") set(DEBUG_BUILD ON) ``` -------------------------------- ### Run Pytest with Tox Source: https://github.com/miurahr/pyppmd/blob/main/docs/contribution.md Execute the test suite using tox for a specific Python version (e.g., py38). This is a convenient way to run tests within the project's defined environments. ```bash tox -e py38 ``` -------------------------------- ### Custom Target for Generating Python Extension Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Defines a custom CMake target to build the Python extension using `setup.py`. This target depends on the virtual environment stamp file and the source files. ```cmake add_custom_target( generate_ext BYPRODUCTS ${PY_EXT} COMMAND ${BUILD_EXT_PYTHON} setup.py build_ext ${BUILD_EXT_OPTION} WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} DEPENDS venv.stamp SOURCES ${pyppmd_sources}) ``` -------------------------------- ### Include Directories for Libraries Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Adds include directories for the project's C/C++ libraries. This allows the compiler to find header files for the buffer and ppmd modules. ```cmake include_directories(src/lib/buffer src/lib/ppmd) ``` -------------------------------- ### Define Source Files for Python Library Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Lists the source files that will be compiled into the `_ppmd` Python module. This includes C files for the extension and the underlying ppmd and buffer libraries. ```cmake set(_sources src/ext/_ppmdmodule.c src/lib/buffer/Buffer.c src/lib/buffer/ThreadDecoder.c src/lib/ppmd/Ppmd7.c src/lib/ppmd/Ppmd7Dec.c src/lib/ppmd/Ppmd7Enc.c src/lib/ppmd/Ppmd8.c src/lib/ppmd/Ppmd8Dec.c src/lib/ppmd/Ppmd8Enc.c) ``` -------------------------------- ### Define Python Extension Paths Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Sets variables for the build output directories and the final locations of the Python extension and CFFI files. These paths are constructed based on build configurations and the determined extension suffix. ```cmake set(PY_BUILD_LIB_DIR build/lib.${PY_PLATFORM}-${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}) set(PY_EXT_INPLACE src/${PY_EXT_DIR}/${PY_EXT_FILE}.${PY_EXT_EXT}) set(PY_EXT ${PY_BUILD_LIB_DIR}/${PY_EXT_DIR}/${PY_EXT_FILE}.${PY_EXT_EXT}) set(PY_CFFI_INPLACE src/${PY_CFFI_DIR}/${PY_CFFI_FILE}.${PY_EXT_EXT}) set(PY_CFFI ${PY_BUILD_LIB_DIR}/${PY_CFFI_DIR}/${PY_CFFI_FILE}.${PY_EXT_EXT}) ``` -------------------------------- ### Compress Data with PyPPMd Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Use the `compress` function for simple, in-memory compression of byte-like objects or strings. Strings are UTF-8 encoded before compression. ```python compressed_data = compress(data) ``` -------------------------------- ### Define pyppmd Library Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Builds the main pyppmd library by compiling C/C++ source files and linking against Python libraries. This is the core library for the project. ```cmake include_directories(src/lib/ppmd src/lib/buffer) add_library( pyppmd src/lib/ppmd/Arch.h src/lib/ppmd/Interface.h src/lib/ppmd/Ppmd.h src/lib/ppmd/Ppmd7.c src/lib/ppmd/Ppmd7.h src/lib/ppmd/Ppmd7Dec.c src/lib/ppmd/Ppmd7Enc.c src/lib/ppmd/Ppmd8.c src/lib/ppmd/Ppmd8.h src/lib/ppmd/Ppmd8Dec.c src/lib/ppmd/Ppmd8Enc.c src/lib/buffer/blockoutput.h src/lib/buffer/Buffer.c src/lib/buffer/Buffer.h src/lib/buffer/win_pthreads.h src/lib/buffer/ThreadDecoder.c src/lib/buffer/ThreadDecoder.h src/ext/_ppmdmodule.c) target_include_directories(pyppmd PRIVATE ${Python_INCLUDE_DIRS}) target_link_libraries(pyppmd PRIVATE ${Python_LIBRARIES}) ``` -------------------------------- ### Set Python Version in CMakeLists.txt Source: https://github.com/miurahr/pyppmd/blob/main/docs/contribution.md Configure the target Python version and implementation for C/C++ bindings development by editing these lines in CMakeLists.txt. ```cmake set(PY_VERSION 3.8) set(Python_FIND_IMPLEMENTATIONS PyPy) ``` -------------------------------- ### Decompress data using pyppmd.decompress() Source: https://context7.com/miurahr/pyppmd/llms.txt Decompresses PPMd-compressed bytes back to bytes. Raises PpmdError on failure. Parameters must match those used during compression. ```python import pyppmd compressed = pyppmd.compress(b"Hello, world! " * 100, max_order=6, mem_size=8 << 20) # Decompress back to bytes result = pyppmd.decompress(compressed, max_order=6, mem_size=8 << 20) assert result == b"Hello, world! " * 100 # Decompress Variant H data compressed_h = pyppmd.compress(b"test data", max_order=6, mem_size=16 << 20, variant="H") result_h = pyppmd.decompress(compressed_h, max_order=6, mem_size=16 << 20, variant="H") assert result_h == b"test data" # Error handling try: bad = pyppmd.decompress(b"\x00\x01\x02", max_order=6, mem_size=8 << 20) except pyppmd.PpmdError as e: print(f"Decompression failed: {e}") ``` -------------------------------- ### Decompress Data to String with PyPPMd Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Use `decompress_str` to decompress data and decode it into a string using a specified encoding (defaults to UTF-8). Raises PpmdError on failure. ```python decompressed_text = decompress_str(data) ``` -------------------------------- ### Streaming Compression with PpmdCompressor Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Utilize the `PpmdCompressor` class for streaming compression. Provide data in chunks using `compress` and finalize with `flush`. The compressor is thread-safe at the method level. ```python c = PpmdCompressor() dat1 = c.compress(b'123456') dat2 = c.compress(b'abcdef') dat3 = c.flush() ``` -------------------------------- ### compress Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Compresses bytes or a string using the PPMd algorithm. Supports custom memory size and variant. ```APIDOC ## compress ### Description Compresses bytes or a string using the PPMd algorithm. Supports custom memory size and variant. ### Parameters * **bytes_or_str** (*bytes-like object* *or* *str*) – Data to be compressed. When it is type of str, encoded with “UTF-8” encoding before compress. * **max_order** (*int*) – maximum order of PPMd algorithm * **mem_size** (*int*) – memory size used for building PPMd model * **variant** (*str*) – PPMd variant name, only accept “H” or “I” ### Returns Compressed data ### Return type bytes ### Example ```python compressed_data = compress(data) ``` ``` -------------------------------- ### Manual Build and Debug C++ Pytest Runner Source: https://github.com/miurahr/pyppmd/blob/main/docs/contribution.md Build the pytest runner executable using CMake and make, then use GDB for C/C++ debugging of tests. This process is useful for debugging C bindings. ```bash mkdir cmake-build cd cmake-build cmake .. make pytest_runner gdb ./pytest_runner ../tests ``` -------------------------------- ### Find Python Interpreter and Development Files Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Locates the Python interpreter and its development components. This is crucial for building Python extensions. ```cmake set(Python_FIND_STRATEGY VERSION) find_package(Python ${PY_VERSION}.0...${PY_VERSION}.99 COMPONENTS Interpreter Development) ``` -------------------------------- ### Set Python Executable for Building Extensions Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Determines the correct Python executable to use for building extensions, accounting for Windows-specific paths and debug builds. This ensures the build process uses the Python interpreter from the virtual environment. ```cmake if (WIN32) if(DEBUG_BUILD) set(BUILD_EXT_PYTHON ${VENV_PATH}/Scripts/python_d.exe) else() set(BUILD_EXT_PYTHON ${VENV_PATH}/bin/python.exe) endif() set(BUILD_EXT_OPTION -g) else() set(BUILD_EXT_PYTHON ${VENV_PATH}/bin/python) set(BUILD_EXT_OPTION -g) endif() ``` -------------------------------- ### Compress data using pyppmd.compress() Source: https://context7.com/miurahr/pyppmd/llms.txt Compresses a bytes-like object or string in a single call. When a str is passed, it is first encoded as UTF-8. Default variant is 'I' (PPMd8). ```python import pyppmd data = b"This file is located in a folder.This file is located in the root." # Compress bytes (Variant I / PPMd8, default) compressed = pyppmd.compress(data, max_order=6, mem_size=8 << 20) print(len(compressed)) # smaller than len(data) for repetitive text # Compress a Python str directly (auto-encoded UTF-8) text = "Hello, PPMd compression!" compressed_str = pyppmd.compress(text, max_order=6, mem_size=8 << 20, variant="I") # Use Variant H (PPMd7, compatible with 7-zip) compressed_h = pyppmd.compress(data, max_order=6, mem_size=16 << 20, variant="H") ``` -------------------------------- ### Configure Tox Test Runner Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Creates a custom CMake target 'run_tox' that executes tox tests within a specified virtual environment. This is used for managing and running project tests across different environments. ```cmake add_custom_target(run_tox COMMAND ${CMAKE_COMMAND} -E env VIRTUAL_ENV=${VENV_PATH} ${VENV_PATH}/bin/python -m tox WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} DEPENDS venv.stamp ) ``` -------------------------------- ### Set C and C++ Standards Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Defines the C and C++ standards to be used for compilation. C11 and C++17 are specified for this project. ```cmake set(CMAKE_C_STANDARD 11) set(CMAKE_CXX_STANDARD 17) ``` -------------------------------- ### Configure Pytest Runner C++ Executable Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Generates a C++ executable that acts as a runner for pytest. It constructs a Python command to append necessary paths and execute pytest with provided arguments. This is useful for integrating C++ build systems with Python testing frameworks. ```cmake file( WRITE ${CMAKE_CURRENT_BINARY_DIR}/pytest_runner.cpp " #include #include #include int main(int argc, char **argv) { std::string args; if ( argc > 1) { args.append(\"[\"); for (int i = 1; i < argc; i++) { if (i > 2) args.append(\",\"); args.append(\"\\\"\"); args.append(argv[i]); args.append(\"\\\"\"); } args.append(\"]\"); } std::filesystem::path src_path = \"${SRC_PATH}\"; std::filesystem::path vsite_path_a = \"${VPKG_PATH_A}\"; std::filesystem::path vsite_path_b = \"${VPKG_PATH_B}\"; std::string pycode = \"import sys\\n\" \"sys.path.append('" + src_path.string() + "')\\n\" \"sys.path.append('" + vsite_path_a.string() + "')\\n\" \"sys.path.append('" + vsite_path_b.string() + "')\\n\" \"import pytest\\n\" \"pytest.main(\" + args + \")\\n\"; execl(\"${Python_EXECUTABLE}\", \"${Python_EXECUTABLE}\", \"-c\", pycode.c_str(), (char*)0); return 0; }" ) add_executable(pytest_runner ${CMAKE_CURRENT_BINARY_DIR}/pytest_runner.cpp) if ("${Python_INTERPRETER_ID}" STREQUAL "PyPy") add_dependencies(pytest_runner generate_cffi) else() add_dependencies(pytest_runner generate_ext build_ext) endif() target_include_directories(pytest_runner PRIVATE ${Python_INCLUDE_DIRS}) target_link_libraries(pytest_runner PRIVATE ${Python_LIBRARIES}) add_custom_target(run_pytest COMMAND $ --verbose WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} DEPENDS pytest_runner) ``` -------------------------------- ### Streaming compression with PpmdCompressor Source: https://context7.com/miurahr/pyppmd/llms.txt A thread-safe streaming compressor. Feed data incrementally with compress() and finalize with flush(). The object cannot be reused after flush() is called. The restore_method parameter only applies to Variant I. ```python import pyppmd # Stream-compress a large file in chunks compressor = pyppmd.PpmdCompressor( max_order=6, mem_size=16 << 20, variant="I", restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) compressed_chunks = [] with open("large_file.txt", "rb") as f: while chunk := f.read(16384): out = compressor.compress(chunk) if out: compressed_chunks.append(out) # Flush remaining buffered data compressed_chunks.append(compressor.flush()) compressed_data = b"".join(compressed_chunks) # Variant H streaming example compressor_h = pyppmd.PpmdCompressor(max_order=6, mem_size=8 << 20, variant="H") part1 = compressor_h.compress(b"first chunk of data") part2 = compressor_h.compress(b"second chunk of data") part3 = compressor_h.flush() result = part1 + part2 + part3 ``` -------------------------------- ### Catching Errors from Streaming Decompressor in PyPPMd Source: https://context7.com/miurahr/pyppmd/llms.txt Catch PyPPMd.PpmdError when using the streaming decompressor, particularly when no data is provided when the decompressor expects input. Initialize the decompressor with appropriate parameters. ```python try: d = pyppmd.PpmdDecompressor(max_order=6, mem_size=8 << 20, variant="I") # Providing no data when decoder needs input raises PpmdError d.decompress(b"") except pyppmd.PpmdError as e: print(f"Stream error: {e}") ``` -------------------------------- ### PpmdCompressor Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md A streaming compressor class for PPMd data. Thread-safe at the method level. ```APIDOC ## PpmdCompressor ### Description A streaming compressor class for PPMd data. Thread-safe at the method level. #### __init__ Initialize a PpmdCompressor object. * **Parameters**: * **max_order** (*int*) – maximum order of PPMd algorithm * **mem_size** (*int*) – memory size used for building PPMd model * **variant** (*str*) – PPMd variant name, only accept “H” or “I” * **restore_method** (*int*) – PPMD8_RESTORE_METHOD_RESTART(0) or PPMD8_RESTORE_METHOD_CUTOFF(1) #### compress Provide data to the compressor object. * **Parameters**: **data** (*bytes-like object*) – Data to be compressed. * **Returns**: A chunk of compressed data if possible, or `b''` otherwise. * **Return type**: bytes #### flush Flush any remaining data in internal buffer. The compressor object can not be used after this method is called. * **Returns**: Flushed data. * **Return type**: bytes ### Example ```python c = PpmdCompressor() dat1 = c.compress(b'123456') dat2 = c.compress(b'abcdef') dat3 = c.flush() ``` ``` -------------------------------- ### compress() — One-shot block compression Source: https://context7.com/miurahr/pyppmd/llms.txt Compresses a bytes-like object or string in a single call using PPMd Variant I or Variant H. Returns the compressed data as bytes. Default variant is 'I' (PPMd8). ```APIDOC ## compress() ### Description Compresses a bytes-like object or string in a single call using PPMd Variant I ("I") or Variant H ("H"). When a `str` is passed it is first encoded as UTF-8. Returns the compressed data as `bytes`. Default variant is `"I"` (PPMd8). ### Parameters - **data** (bytes or str) - The data to compress. - **max_order** (int) - Optional. The maximum order for the PPMd model. Defaults to 6. - **mem_size** (int) - Optional. The memory size to allocate for the PPMd model. Defaults to 8 << 20. - **variant** (str) - Optional. The PPMd variant to use, either "I" or "H". Defaults to "I". ### Returns - **bytes** - The compressed data. ### Request Example ```python import pyppmd data = b"This file is located in a folder.This file is located in the root." # Compress bytes (Variant I / PPMd8, default) compressed = pyppmd.compress(data, max_order=6, mem_size=8 << 20) print(len(compressed)) # smaller than len(data) for repetitive text # Compress a Python str directly (auto-encoded UTF-8) text = "Hello, PPMd compression!" compressed_str = pyppmd.compress(text, max_order=6, mem_size=8 << 20, variant="I") # Use Variant H (PPMd7, compatible with 7-zip) compressed_h = pyppmd.compress(data, max_order=6, mem_size=16 << 20, variant="H") ``` ``` -------------------------------- ### Decompress Data to Bytes with PyPPMd Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Use the `decompress` function for simple, in-memory decompression of byte-like objects. Raises PpmdError on failure. ```python decompressed_data = decompress(data) ``` -------------------------------- ### Low-level PPMd Variant H (7-zip) Encoding and Decoding Source: https://context7.com/miurahr/pyppmd/llms.txt Ppmd7Encoder and Ppmd7Decoder handle PPMd Variant H with 7-zip's range coder, suitable for py7zr. `max_order` must be 2-64. Be aware of a potential trailing null byte omission; supply `b"\0"` if decoded length is short. ```python import pyppmd data = b"This file is located in a folder.This file is located in the root." # Encode in one shot encoder = pyppmd.Ppmd7Encoder(max_order=6, mem_size=16 << 20) compressed = encoder.encode(data) compressed += encoder.flush() # endmark=False by default # compressed += encoder.flush(endmark=True) # write end marker (-1) # Encode incrementally encoder2 = pyppmd.Ppmd7Encoder(6, 16 << 20) compressed2 = encoder2.encode(data[:33]) compressed2 += encoder2.encode(data[33:]) compressed2 += encoder2.flush(endmark=False) # Decode — must supply expected output length decoder = pyppmd.Ppmd7Decoder(max_order=6, mem_size=16 << 20) result = decoder.decode(compressed, length=len(data)) # Handle the "extra null byte" edge case if len(result) < len(data): if decoder.needs_input: result += decoder.decode(b"\0", len(data) - len(result)) else: result += decoder.decode(b"", len(data) - len(result)) assert result == data print(decoder.eof) # True when end of stream reached print(decoder.needs_input) # False when internal buffer still has data ``` -------------------------------- ### Add Python Library Target Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Creates a CMake target for the Python extension module `_ppmd`. It specifies that this is a shared library with the Python ABI. ```cmake Python_add_library(_ppmd MODULE WITH_SOABI ${_sources}) ``` -------------------------------- ### decompress Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Decompresses data and returns it as bytes. ```APIDOC ## decompress ### Description Decompresses data and returns it as bytes. ### Parameters * **data** (*bytes-like object*) – Data to be decompressed * **max_order** (*int*) – maximum order of PPMd algorithm * **mem_size** (*int*) – memory size used for building PPMd model * **variant** (*str*) – PPMd variant name, only accept “H” or “I” ### Returns Decompressed data ### Return type bytes ### Raises [**PpmdError**](#PpmdError) – If decompression fails. ### Example ```python decompressed_data = decompress(data) ``` ``` -------------------------------- ### decompress_str Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Decompresses data and returns it as a string, with support for custom encoding. ```APIDOC ## decompress_str ### Description Decompresses data and returns it as a string, with support for custom encoding. ### Parameters * **data** (*bytes-like object*) – Data to be decompressed. * **max_order** (*int*) – maximum order of PPMd algorithm * **mem_size** (*int*) – memory size used for building PPMd model * **encoding** (*str*) – Encoding name to use when decoding raw decompressed data * **variant** (*str*) – PPMd variant name, only accept “H” or “I” ### Returns Decompressed text ### Return type str ### Raises [**PpmdError**](#PpmdError) – If decompression fails. ### Example ```python decompress_text = decompress_str(data) ``` ``` -------------------------------- ### Custom Target for Generating CFFI Extension Source: https://github.com/miurahr/pyppmd/blob/main/CMakeLists.txt Defines a custom CMake target to build the CFFI Python extension. This target is similar to `generate_ext` but includes the `--cffi --inplace` options for CFFI builds. ```cmake add_custom_target( generate_cffi BYPRODUCTS ${PY_CFFI_INPLACE} COMMAND ${BUILD_EXT_PYTHON} setup.py build_ext ${BUILD_EXT_OPTION} --cffi --inplace WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} DEPENDS venv.stamp SOURCES ${pyppmd_sources}) ``` -------------------------------- ### Ppmd8Encoder / Ppmd8Decoder Source: https://context7.com/miurahr/pyppmd/llms.txt Low-level classes for PPMd Variant I version 2, designed for general-purpose text compression. Offers control over model recovery during decompression. ```APIDOC ## `Ppmd8Encoder` / `Ppmd8Decoder` — Low-level PPMd Variant I version 2 The `Ppmd8Encoder` and `Ppmd8Decoder` classes implement PPMd Variant I version 2 for general-purpose text compression. The `restore_method` controls model recovery: `PPMD8_RESTORE_METHOD_RESTART` (0) resets the model, `PPMD8_RESTORE_METHOD_CUT_OFF` (1) cuts off the context. Both encoder and decoder must use the same parameters. ### Encoding and Decoding ```python import pyppmd source = b"This file is located in a folder.This file is located in the root.\n" # --- Encoding --- encoder = pyppmd.Ppmd8Encoder( max_order=6, mem_size=8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) compressed = encoder.encode(source[:33]) compressed += encoder.encode(source[33:]) compressed += encoder.flush(endmark=True) # writes end marker; default is True # --- Decoding --- decoder = pyppmd.Ppmd8Decoder( max_order=6, mem_size=8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) result = decoder.decode(compressed, -1) # length=-1 means unlimited output result += decoder.decode(b"", -1) # drain remaining buffered data assert result == source # --- Stream file compression/decompression --- import hashlib READ_BLOCKSIZE = 16384 enc = pyppmd.Ppmd8Encoder(6, 8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF) compressed_data = b"" with open("input.txt", "rb") as f: while chunk := f.read(READ_BLOCKSIZE): compressed_data += enc.encode(chunk) compressed_data += enc.flush(endmark=True) dec = pyppmd.Ppmd8Decoder(6, 8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF) decompressed = b"" while not dec.eof: chunk = compressed_data[:READ_BLOCKSIZE] compressed_data = compressed_data[READ_BLOCKSIZE:] decompressed += dec.decode(chunk) if not compressed_data: break ``` ``` -------------------------------- ### Catching Errors from One-Shot Decompression in PyPPMd Source: https://context7.com/miurahr/pyppmd/llms.txt Use a try-except block to catch PyPPMd.PpmdError when decompressing data with the one-shot decompress function. Ensure correct parameters like max_order and mem_size are provided. ```python try: result = pyppmd.decompress(b"\xff\xfe\xfd", max_order=6, mem_size=8 << 20) except pyppmd.PpmdError as e: print(f"Decompression error: {e}") ``` -------------------------------- ### PpmdDecompressor Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md A streaming decompressor class for PPMd data. Thread-safe at the method level. ```APIDOC ## PpmdDecompressor ### Description A streaming decompressor class for PPMd data. Thread-safe at the method level. #### __init__ Initialize a PpmdDecompressor object. * **Parameters**: * **max_order** (*int*) – maximum order of PPMd algorithm * **mem_size** (*int*) – memory size used for building PPMd model * **variant** (*str*) – PPMd variant name, only accept “H” or “I” * **restore_method** (*int*) – PPMD8_RESTORE_METHOD_RESTART(0) or PPMD8_RESTORE_METHOD_CUTOFF(1) #### decompress Decompress *data*, returning decompressed data as a `bytes` object. * **Parameters**: * **data** (*bytes-like object*) – Data to be decompressed. * **max_length** (*int*) – Maximum size of returned data. When it’s negative, the output size is unlimited. When it’s non-negative, returns at most *max_length* bytes of decompressed data. If this limit is reached and further output can (or may) be produced, the [`needs_input`](#PpmdDecompressor.needs_input) attribute will be set to `False`. In this case, the next call to this method may provide *data* as `b''` to obtain more of the output. ``` -------------------------------- ### Low-level PPMd Variant I version 2 Encoding and Decoding Source: https://context7.com/miurahr/pyppmd/llms.txt Ppmd8Encoder and Ppmd8Decoder implement PPMd Variant I version 2 for general text compression. Ensure both encoder and decoder use identical parameters. `restore_method` controls model recovery (restart or cut-off). ```python import pyppmd source = b"This file is located in a folder.This file is located in the root.\n" # --- Encoding --- encoder = pyppmd.Ppmd8Encoder( max_order=6, mem_size=8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) compressed = encoder.encode(source[:33]) compressed += encoder.encode(source[33:]) compressed += encoder.flush(endmark=True) # writes end marker; default is True # --- Decoding --- decoder = pyppmd.Ppmd8Decoder( max_order=6, mem_size=8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) result = decoder.decode(compressed, -1) # length=-1 means unlimited output result += decoder.decode(b"", -1) # drain remaining buffered data assert result == source # --- Stream file compression/decompression --- import hashlib READ_BLOCKSIZE = 16384 enc = pyppmd.Ppmd8Encoder(6, 8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF) compressed_data = b"" with open("input.txt", "rb") as f: while chunk := f.read(READ_BLOCKSIZE): compressed_data += enc.encode(chunk) compressed_data += enc.flush(endmark=True) dec = pyppmd.Ppmd8Decoder(6, 8 << 20, restore_method=pyppmd.PPMD8_RESTORE_METHOD_CUT_OFF) decompressed = b"" while not dec.eof: chunk = compressed_data[:READ_BLOCKSIZE] compressed_data = compressed_data[READ_BLOCKSIZE:] decompressed += dec.decode(chunk) if not compressed_data: break ``` -------------------------------- ### PpmdError Source: https://context7.com/miurahr/pyppmd/llms.txt The exception class raised for any failures during compression or decompression operations, such as corrupt input, insufficient data, or memory allocation issues. ```APIDOC ## `PpmdError` — Exception class Raised when a compression or decompression operation fails (e.g., corrupt input data, insufficient input, or memory allocation failure). ### Usage ```python import pyppmd # Example of catching PpmdError (specific usage depends on context) try: # Perform a compression or decompression operation that might fail pass except pyppmd.PpmdError as e: print(f"PPMD operation failed: {e}") ``` ``` -------------------------------- ### Decompressing Data in Chunks Source: https://github.com/miurahr/pyppmd/blob/main/docs/api_guide.md Use this pattern when decompressing data that may exceed the maximum output length in a single call. If `needs_input` is True after a decompress call, pass empty bytes (`b''`) to continue decompressing. ```python d1 = PpmdDecompressor() decompressed_dat = d1.decompress(dat1) decompressed_dat += d1.decompress(dat2) decompressed_dat += d1.decompress(dat3) ``` -------------------------------- ### decompress() — One-shot block decompression Source: https://context7.com/miurahr/pyppmd/llms.txt Decompresses a PPMd-compressed bytes-like object back to bytes. Raises PpmdError on failure. The variant, max_order, and mem_size parameters must match those used during compression. ```APIDOC ## decompress() ### Description Decompresses a PPMd-compressed bytes-like object back to `bytes`. Raises `PpmdError` on failure. The `variant`, `max_order`, and `mem_size` parameters must match those used during compression. ### Parameters - **compressed_data** (bytes) - The compressed data to decompress. - **max_order** (int) - Optional. The maximum order used during compression. Defaults to 6. - **mem_size** (int) - Optional. The memory size used during compression. Defaults to 8 << 20. - **variant** (str) - Optional. The PPMd variant used during compression, either "I" or "H". Defaults to "I". ### Returns - **bytes** - The decompressed data. ### Raises - **PpmdError** - If decompression fails. ### Request Example ```python import pyppmd compressed = pyppmd.compress(b"Hello, world! " * 100, max_order=6, mem_size=8 << 20) # Decompress back to bytes result = pyppmd.decompress(compressed, max_order=6, mem_size=8 << 20) assert result == b"Hello, world! " * 100 # Decompress Variant H data compressed_h = pyppmd.compress(b"test data", max_order=6, mem_size=16 << 20, variant="H") result_h = pyppmd.decompress(compressed_h, max_order=6, mem_size=16 << 20, variant="H") assert result_h == b"test data" # Error handling try: bad = pyppmd.decompress(b"\x00\x01\x02", max_order=6, mem_size=8 << 20) except pyppmd.PpmdError as e: print(f"Decompression failed: {e}") ``` ``` -------------------------------- ### PpmdDecompressor Source: https://context7.com/miurahr/pyppmd/llms.txt A thread-safe streaming decompressor class. It handles feeding compressed data in chunks and provides attributes to monitor the stream's end, input needs, and unused data. ```APIDOC ## `PpmdDecompressor` — Streaming decompressor class A thread-safe streaming decompressor. The `eof` attribute signals the end of a compressed stream. The `needs_input` attribute indicates whether more compressed bytes are required. The `unused_data` attribute holds any bytes after the end marker. ### Usage ```python import pyppmd compressed = pyppmd.compress(b"Hello streaming world! " * 200, max_order=6, mem_size=8 << 20) decompressor = pyppmd.PpmdDecompressor( max_order=6, mem_size=8 << 20, variant="I", restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) # Feed compressed data in pieces output = b"" chunk_size = 20 for i in range(0, len(compressed), chunk_size): chunk = compressed[i:i + chunk_size] output += decompressor.decompress(chunk) if decompressor.eof: break # Drain any remaining buffered output if not decompressor.needs_input: output += decompressor.decompress(b"") assert output == b"Hello streaming world! " * 200 # Bounded decompression with max_length decompressor2 = pyppmd.PpmdDecompressor(max_order=6, mem_size=8 << 20, variant="I") partial = decompressor2.decompress(compressed, max_length=50) # If output was truncated, needs_input is False — pass b"" to get more if not decompressor2.needs_input: partial += decompressor2.decompress(b"") ``` ``` -------------------------------- ### decompress_str() — One-shot decompression to string Source: https://context7.com/miurahr/pyppmd/llms.txt Decompresses PPMd-compressed data and decodes the result to a Python str using a specified encoding (default UTF-8). Useful when the original data was text. Raises PpmdError on failure. ```APIDOC ## decompress_str() ### Description Decompresses PPMd-compressed data and decodes the result to a Python `str` using a specified encoding (default UTF-8). Useful when the original data was text. Raises `PpmdError` on failure. ### Parameters - **compressed_data** (bytes) - The compressed data to decompress. - **max_order** (int) - Optional. The maximum order used during compression. Defaults to 6. - **mem_size** (int) - Optional. The memory size used during compression. Defaults to 8 << 20. - **encoding** (str) - Optional. The encoding to use for decoding the decompressed data. Defaults to "UTF-8". - **variant** (str) - Optional. The PPMd variant used during compression, either "I" or "H". Defaults to "I". ### Returns - **str** - The decompressed and decoded string. ### Raises - **PpmdError** - If decompression fails. ### Request Example ```python import pyppmd original = "Unicode text: café, naïve, 日本語" compressed = pyppmd.compress(original, max_order=6, mem_size=8 << 20) # Decompress and decode to str text = pyppmd.decompress_str(compressed, max_order=6, mem_size=8 << 20, encoding="UTF-8") assert text == original print(text) # Unicode text: café, naïve, 日本語 # Using a different source encoding latin_data = "Ångström".encode("latin-1") compressed_latin = pyppmd.compress(latin_data, max_order=6, mem_size=8 << 20) recovered = pyppmd.decompress_str(compressed_latin, max_order=6, mem_size=8 << 20, encoding="latin-1") assert recovered == "Ångström" ``` ``` -------------------------------- ### PpmdCompressor — Streaming compressor class Source: https://context7.com/miurahr/pyppmd/llms.txt A thread-safe streaming compressor. Feed data incrementally with compress() and finalize with flush(). The object cannot be reused after flush() is called. The restore_method parameter only applies to Variant I. ```APIDOC ## PpmdCompressor ### Description A thread-safe streaming compressor. Feed data incrementally with `compress()` and finalize with `flush()`. The object cannot be reused after `flush()` is called. The `restore_method` parameter only applies to Variant I. ### Parameters - **max_order** (int) - Optional. The maximum order for the PPMd model. Defaults to 6. - **mem_size** (int) - Optional. The memory size to allocate for the PPMd model. Defaults to 8 << 20. - **variant** (str) - Optional. The PPMd variant to use, either "I" or "H". Defaults to "I". - **restore_method** (int) - Optional. The restore method for Variant I. Defaults to `pyppmd.PPMD8_RESTORE_METHOD_RESTART`. ### Methods - **compress(data: bytes)**: Compresses a chunk of data and returns any compressed output. Can be called multiple times. - **flush()**: Flushes any remaining buffered data and finalizes the compression stream. Returns the remaining compressed data. This method can only be called once. ### Request Example ```python import pyppmd # Stream-compress a large file in chunks compressor = pyppmd.PpmdCompressor( max_order=6, mem_size=16 << 20, variant="I", restore_method=pyppmd.PPMD8_RESTORE_METHOD_RESTART, ) compressed_chunks = [] with open("large_file.txt", "rb") as f: while chunk := f.read(16384): out = compressor.compress(chunk) if out: compressed_chunks.append(out) # Flush remaining buffered data compressed_chunks.append(compressor.flush()) compressed_data = b"".join(compressed_chunks) # Variant H streaming example compressor_h = pyppmd.PpmdCompressor(max_order=6, mem_size=8 << 20, variant="H") part1 = compressor_h.compress(b"first chunk of data") part2 = compressor_h.compress(b"second chunk of data") part3 = compressor_h.flush() result = part1 + part2 + part3 ``` ``` -------------------------------- ### Ppmd7Encoder / Ppmd7Decoder Source: https://context7.com/miurahr/pyppmd/llms.txt Low-level classes for PPMd Variant H (7-zip) compression and decompression, utilizing 7-zip's range coder. Suitable for integration with 7-zip archive libraries. ```APIDOC ## `Ppmd7Encoder` / `Ppmd7Decoder` — Low-level PPMd Variant H (7-zip) The `Ppmd7Encoder` and `Ppmd7Decoder` classes operate on PPMd Variant H with 7-zip's range coder. They are designed for use in 7-zip archive libraries (e.g., py7zr). `max_order` must be between 2 and 64. The PPMd algorithm may omit a trailing null byte; supply `b"\0"` as an extra input byte if the decoded length falls short. ### Usage ```python import pyppmd data = b"This file is located in a folder.This file is located in the root." # Encode in one shot encoder = pyppmd.Ppmd7Encoder(max_order=6, mem_size=16 << 20) compressed = encoder.encode(data) compressed += encoder.flush() # endmark=False by default # compressed += encoder.flush(endmark=True) # write end marker (-1) # Encode incrementally encoder2 = pyppmd.Ppmd7Encoder(6, 16 << 20) compressed2 = encoder2.encode(data[:33]) compressed2 += encoder2.encode(data[33:]) compressed2 += encoder2.flush(endmark=False) # Decode — must supply expected output length decoder = pyppmd.Ppmd7Decoder(max_order=6, mem_size=16 << 20) result = decoder.decode(compressed, length=len(data)) # Handle the "extra null byte" edge case if len(result) < len(data): if decoder.needs_input: result += decoder.decode(b"\0", len(data) - len(result)) else: result += decoder.decode(b"", len(data) - len(result)) assert result == data print(decoder.eof) # True when end of stream reached print(decoder.needs_input) # False when internal buffer still has data ``` ```