### ZstdCompressor Usage Examples Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Examples demonstrating how to use the ZstdCompressor for streaming compression. ```APIDOC c = ZstdCompressor() # traditional streaming compression dat1 = c.compress(b'123456') dat2 = c.compress(b'abcdef') dat3 = c.flush() # use .compress() method with mode argument compressed_dat1 = c.compress(raw_dat1, c.FLUSH_BLOCK) compressed_dat2 = c.compress(raw_dat2, c.FLUSH_FRAME) ``` -------------------------------- ### ZstdCompressor Usage Example Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates basic compression and flushing using the ZstdCompressor object. Ensure raw_dat is defined before use. ```python c = ZstdCompressor(level_or_option=option) compressed_dat1 = c.compress(raw_dat) compressed_dat2 = c.flush() ``` -------------------------------- ### Streaming Decompression with Limited Output Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Use EndlessZstdDecompressor for streaming decompression when the output size is unknown or potentially very large. This example limits the output buffer to prevent excessive memory usage. ```python d2 = EndlessZstdDecompressor() while True: if d2.needs_input: dat = read_input(2*1024*1024) # read 2 MiB input data if not dat: # input stream ends if not d2.at_frame_edge: raise Exception('data ends in an incomplete frame.') break else: # maybe there is unconsumed input data dat = b'' chunk = d2.decompress(dat, 10*1024*1024) # limit output buffer to 10 MiB write_output(chunk) ``` -------------------------------- ### Get Zstandard Frame Information Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Use get_frame_info to extract decompressed size and dictionary ID from a zstd frame header. Ensure the input buffer starts at the frame beginning and contains at least the header. ```python >>> pyzstd.get_frame_info(compressed_dat[:20]) frame_info(decompressed_size=687379, dictionary_id=1040992268) ``` -------------------------------- ### Compression with Strategy and Checksum Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Shows how to configure compression using `CParameter.strategy` and `CParameter.checksumFlag`. ```APIDOC ## Compression with Strategy and Checksum ### Description Configure compression parameters, including the compression strategy and whether to include a checksum. ### Usage ```python option = {CParameter.strategy : Strategy.lazy2, CParameter.checksumFlag : 1} compressed_dat = compress(raw_dat, option=option) ``` ### Parameters #### `CParameter.strategy` - **Type**: `Strategy` (IntEnum) - **Description**: Used for compression strategies, listed from fastest to strongest. New strategies might be added in the future, only the order (from fast to strong) is guaranteed. #### `CParameter.checksumFlag` - **Type**: Integer (0 or 1) - **Description**: Flag to indicate whether to include a checksum in the compressed data. ``` -------------------------------- ### Get Zstandard Compression Level Values Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Access compressionLevel_values to get the default, minimum, and maximum compression levels supported by the underlying zstd library. ```python >>> pyzstd.compressionLevel_values # 131072 = 128*1024 values(default=3, min=-131072, max=22) ``` -------------------------------- ### Set Advanced Compression Parameters Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates setting advanced compression parameters using a dictionary with CParameter enums. Used with the compress() function. ```python option = {CParameter.compressionLevel : 10, CParameter.checksumFlag : 1} # used with compress() function compressed_dat = compress(raw_dat, option) ``` -------------------------------- ### Accessing Compression Parameter Bounds Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Shows how to retrieve the inclusive lower and upper bounds for compression parameters like compressionLevel, windowLog, and enableLongDistanceMatching. ```python CParameter.compressionLevel.bounds() CParameter.windowLog.bounds() CParameter.enableLongDistanceMatching.bounds() ``` -------------------------------- ### Get Zstandard Library Version (String) Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Access the zstd_version variable to retrieve the version of the underlying zstd library as a string. ```python >>> pyzstd.zstd_version '1.4.5' ``` -------------------------------- ### Get Zstandard Library Version (Tuple) Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Access the zstd_version_info variable to retrieve the version of the underlying zstd library as a tuple of integers. ```python >>> pyzstd.zstd_version_info (1, 4, 5) ``` -------------------------------- ### Load and Use Zstd Dictionary for Compression/Decompression Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Demonstrates loading a zstd dictionary from a file and using it with `compress` and `decompress` functions. Reusing compressor objects can improve performance when applying the same dictionary multiple times. ```python import io from pyzstd import ZstdDict, compress, decompress # load a zstd dictionary from file with io.open(dict_path, 'rb') as f: file_content = f.read() zd = ZstdDict(file_content) # use the dictionary to compress. # if use a dictionary for compressor multiple times, reusing # a compressor object is faster, see .as_undigested_dict doc. compressed_dat = compress(raw_dat, zstd_dict=zd) # use the dictionary to decompress decompressed_dat = decompress(compressed_dat, zstd_dict=zd) ``` -------------------------------- ### Get Compression Level Bounds Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Retrieves the inclusive lower and upper bounds for the compression level parameter using the bounds() method of the CParameter enum. ```python >>> CParameter.compressionLevel.bounds() (-131072, 22) ``` -------------------------------- ### Replace compress_stream with pyzstd.open Source: https://pyzstd.readthedocs.io/en/latest/_sources/deprecated.md.txt Use `pyzstd.open` for stream compression instead of `compress_stream`. For more control, consider `ZstdCompressor`. ```python # before with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: compress_stream(ifh, ofh, level_or_option=5) # after with io.open(input_file_path, 'rb') as ifh: with pyzstd.open(output_file_path, 'w', level_or_option=5) as ofh: shutil.copyfileobj(ifh, ofh) ``` ```python # after: more complex alternative with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: compressor = ZstdCompressor(level_or_option=5) compressor._set_pledged_input_size(pledged_input_size) # optional while data := ifh.read(read_size): ofh.write(compressor.compress(data)) callback_progress(ifh.tell(), ofh.tell()) # optional ofh.write(compressor.flush()) ``` -------------------------------- ### Initialize ZstdCompressor Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Initialize a ZstdCompressor object. The compression level or advanced parameters can be specified, or a pre-trained dictionary can be provided. Defaults to zstd's default compression level. ```python c = ZstdCompressor() ``` -------------------------------- ### Get Bounds for DParameter.windowLogMax Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Retrieve the lower and upper inclusive bounds for the DParameter.windowLogMax parameter. This helps in understanding the valid range for memory allocation limits. ```python DParameter.windowLogMax.bounds() ``` -------------------------------- ### Initialize ZstdCompressor Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Create a `ZstdCompressor` object for streaming compression. This object can be initialized with a compression level (integer) or a dictionary of advanced compression parameters. ```python compressor = ZstdCompressor(level_or_option=None, zstd_dict=None) ``` -------------------------------- ### Compress and Decompress Tar Archives with Zstd Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Demonstrates how to open and interact with .tar.zst archives for both writing (compression) and reading (decompression) using a custom TarFile implementation. ```python with CustomTarFile.open('archive.tar.zst', mode='w:zst') as tar: # do something with CustomTarFile.open('archive.tar.zst') as tar: # do something ``` -------------------------------- ### Get Decompression Parameter Bounds Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt The bounds() method of a DParameter enum member returns the inclusive lower and upper bounds for its valid values. This helps in understanding the acceptable range for specific decompression settings. ```python >>> DParameter.windowLogMax.bounds() (10, 31) ``` -------------------------------- ### Replace compress_stream with pyzstd.open Source: https://pyzstd.readthedocs.io/en/latest/deprecated.html Use `pyzstd.open` for stream compression instead of `compress_stream`. For more control, consider using `ZstdCompressor`. ```python # before with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: compress_stream(ifh, ofh, level_or_option=5) ``` ```python # after with io.open(input_file_path, 'rb') as ifh: with pyzstd.open(output_file_path, 'w', level_or_option=5) as ofh: shutil.copyfileobj(ifh, ofh) ``` ```python # after: more complex alternative with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: compressor = ZstdCompressor(level_or_option=5) compressor._set_pledged_input_size(pledged_input_size) # optional while data := ifh.read(read_size): ofh.write(compressor.compress(data)) callback_progress(ifh.tell(), ofh.tell()) # optional ofh.write(compressor.flush()) ``` -------------------------------- ### Get Zstandard Frame Size Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Use get_frame_size to determine the total size of a zstd frame, including header and optional checksum. The input buffer must contain at least one complete frame. ```python >>> pyzstd.get_frame_size(compressed_dat) 252874 ``` -------------------------------- ### hashLog Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Sets the size of the initial probe table as a power of 2. Affects compression ratio and speed. ```APIDOC ## hashLog ### Description Determines the size of the initial probe table, expressed as a power of 2. The resulting memory usage is `1 << (hashLog + 2)` bytes. Larger tables can improve compression ratio for certain strategies and speed for others. ### Parameter Details - **hashLog**: Size of the initial probe table, as a power of 2. Memory usage is `1 << (hashLog + 2)` bytes. Must be clamped between `ZSTD_HASHLOG_MIN` and `ZSTD_HASHLOG_MAX`. Larger tables improve compression ratio of strategies <= `dfast`, and improve speed of strategies > `dfast`. Special value `0` means “use default hashLog”, then the value is dynamically set. ### Bounds Refer to `CParameter.hashLog.bounds()` for specific bounds. ``` -------------------------------- ### Set Compression Strategy and Checksum Flag Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Configure compression using CParameter.strategy and CParameter.checksumFlag. Strategy.lazy2 is used for a balance between speed and compression ratio, and checksumFlag enables data integrity checks. ```python option = {CParameter.strategy : Strategy.lazy2, CParameter.checksumFlag : 1} compressed_dat = compress(raw_dat, option) ``` -------------------------------- ### Streaming Compression with ZstdCompressor Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Demonstrates traditional streaming compression using ZstdCompressor. Data is fed in chunks, and the compressor manages frame and block flushing. ```python c = ZstdCompressor() # traditional streaming compression dat1 = c.compress(b'123456') dat2 = c.compress(b'abcdef') dat3 = c.flush() ``` -------------------------------- ### Traditional Streaming Compression with ZstdCompressor Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates traditional streaming compression using the ZstdCompressor. Data is fed in chunks using the compress() method, and remaining data is flushed using flush(). ```python # traditional streaming compression dat1 = c.compress(b'123456') dat2 = c.compress(b'abcdef') dat3 = c.flush() ``` -------------------------------- ### Decompression with Memory Limit Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates how to set a memory allocation limit for decompression using `DParameter.windowLogMax`. This is useful when dealing with frames that might require more memory than desired. ```APIDOC ## Decompression with Memory Limit ### Description Set a memory allocation limit (in power of 2) for decompression to protect the host from unreasonable memory requirements. If a frame requires more memory than the set value, a `ZstdError` exception is raised. ### Usage ```python # Set memory allocation limit to 16 MiB (1 << 24) option = {DParameter.windowLogMax : 24} # Used with decompress() function decompressed_dat = decompress(dat, option=option) # Used with ZstdDecompressor object d = ZstdDecompressor(option=option) decompressed_dat = d.decompress(dat) ``` ### Parameters #### `DParameter.windowLogMax` - **Type**: Integer - **Description**: Select a size limit (in power of 2) beyond which the streaming API will refuse to allocate memory buffer. Special value `0` means "use default maximum windowLog". By default, accepted window sizes are <= `(1 << 27)` (128 MiB). If a frame requests a window size greater than this value, this parameter needs to be explicitly set. ``` -------------------------------- ### Compressing data with a prefix for patching Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Use VER_1 as a prefix to compress VER_2, enabling efficient patching. Ensure windowLog is set appropriately and long distance matching is enabled. The resulting PATCH is smaller than the full VER_2. ```python # use VER_1 as prefix v1 = ZstdDict(VER_1, is_raw=True) # let the window cover the longest version. # don't forget to clamp windowLog to valid range. # enable "long distance matching". windowLog = max(len(VER_1), len(VER_2)).bit_length() option = {CParameter.windowLog: windowLog, CParameter.enableLongDistanceMatching: 1} # get a small PATCH PATCH = compress(VER_2, level_or_option=option, zstd_dict=v1.as_prefix) ``` -------------------------------- ### CParameter Usage Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates how to use the CParameter enum to set advanced compression options. Parameters are provided as a dictionary where keys are CParameter enum members and values are integers. ```APIDOC ## CParameter ### Description Advanced compression parameters. When using, put the parameters in a `dict` object, the key is a `CParameter` name, the value is a 32-bit signed integer value. ### Example ```python option = {CParameter.compressionLevel : 10, CParameter.checksumFlag : 1} # used with compress() function compressed_dat = compress(raw_dat, option) ``` ``` -------------------------------- ### ZstdFile Initialization Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Initializes a ZstdFile object for reading, writing, or appending. The `filename` argument can be a file path or a file object. Advanced compression parameters can be specified using `level_or_option` and `zstd_dict`. `max_frame_content_size` is crucial for compression modes to control frame size and seeking performance. ```APIDOC ## ZstdFile.__init__ ### Description Initializes a ZstdFile object. For append modes ('a', 'ab'), `filename` must be a file path. ### Parameters * **filename** (str/bytes/PathLike or file object) - The file to open. * **mode** (str) - The mode to open the file in (e.g., 'r', 'w', 'a', 'rb', 'wb', 'ab'). Defaults to 'r'. * **level_or_option** (int or dict) - Compression level or advanced compression options. * **zstd_dict** (ZstdDict) - A dictionary for compression/decompression. * **max_frame_content_size** (int) - Maximum size of a compression frame. Used in compression modes. ### Attention * `max_frame_content_size` is used for compression modes ('w', 'wb', 'a', 'ab', 'x', 'xb'). * When uncompressed data reaches this size, the current frame is automatically closed. * Default value (1 GiB) is often not optimal; user should set based on data and seeking needs. * Smaller frame sizes improve seeking speed but reduce compression ratio. Larger sizes do the opposite. * Avoid very small frame sizes (<1 KiB) as they significantly hurt compression ratio. * Frames can be manually closed using `f.flush(mode=f.FLUSH_FRAME)`. ``` -------------------------------- ### Set Advanced Compression Options Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Demonstrates how to set advanced compression parameters using the CParameter enum. These options can be passed to the compress() function or a ZstdCompressor object. ```python option = {CParameter.compressionLevel : 10, CParameter.checksumFlag : 1} # used with compress() function compressed_dat = compress(raw_dat, option) # used with ZstdCompressor object c = ZstdCompressor(level_or_option=option) compressed_dat1 = c.compress(raw_dat) compressed_dat2 = c.flush() ``` -------------------------------- ### Compression Level Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Explains the different compression levels available, including regular, ultra, default, and negative levels. ```APIDOC ## Compression Level ### Description Compression level is an integer that determines the trade-off between compression speed and ratio. ### Levels - **Regular Levels**: `1` to `22` (currently). Levels >= 20, labeled _ultra_, should be used with caution as they require more memory. - **Default Level**: `0` means use the default level, which is currently `3`. - **Negative Levels**: `-131072` to `-1`. Lower levels mean faster speed but lower compression ratio. ### Advanced Usage Compression levels map to a set of compression parameters. Setting a compression level may dynamically impact other un-set compression parameters. Manually set parameters will remain. ### Note `compressionLevel_values` are defined by the underlying zstd library. For advanced users, compression levels are numbers that map to a set of compression parameters. The parameters may be adjusted by the underlying zstd library after gathering information, such as data size, using a dictionary or not. Setting a compression level does not set all other compression parameters to default. Setting this will dynamically impact the compression parameters which have not been manually set, the manually set ones will “stick”. ``` -------------------------------- ### compress Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Compresses data using the Zstandard algorithm. It can accept a compression level (int) or advanced parameters (dict). Optionally, a pre-trained dictionary can be provided for better compression ratios. ```APIDOC ## compress(data, level_or_option=None, zstd_dict=None) ### Description Compress *data*, return the compressed data. Compressing ``b''`` will get an empty content frame (9 bytes or more). ### Parameters * **data** (bytes-like object) - Data to be compressed. * **level_or_option** (int or dict) - When it's an ``int`` object, it represents :ref:`compression level`. When it's a ``dict`` object, it contains :ref:`advanced compression parameters`. The default value ``None`` means to use zstd's default compression level/parameters. * **zstd_dict** (ZstdDict) - Pre-trained dictionary for compression. ### Returns * **bytes** - Compressed data ### Examples ```python # int compression level compressed_dat = compress(raw_dat, 10) # dict option, use 6 threads to compress, and append a 4-byte checksum. option = { CParameter.compressionLevel : 10, CParameter.nbWorkers : 6, CParameter.checksumFlag : 1 } compressed_dat = compress(raw_dat, option) ``` ``` -------------------------------- ### open function Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Opens a zstd-compressed file, supporting both binary and text modes. ```APIDOC ## open(filename, mode='r', ...) ### Description Opens a zstd-compressed file in binary or text mode. This function acts as a convenient wrapper for creating or accessing Zstandard compressed files, similar to Python's built-in `open()` function. ### Parameters - **filename**: The path to the file to be opened (string, bytes, or path-like object). - **mode**: The mode in which to open the file. Supports 'r', 'w', 'x', 'a' for reading, writing, exclusive creation, and appending, respectively. Can be combined with 'b' for binary mode (e.g., 'rb', 'wb') or 't' for text mode (e.g., 'rt', 'wt'). Defaults to 'r'. - **level_or_option**: Compression level or option, used when writing. - **zstd_dict**: A Zstandard dictionary for compression or decompression. ### Returns A file-like object (instance of `ZstdFile` or a similar object for text mode) that allows reading from or writing to the compressed file. ``` -------------------------------- ### Decompressing data with a prefix for patching Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Decompress a PATCH using the same prefix (VER_1) used during compression. Explicitly set windowLogMax if the required window size exceeds 128MiB. Failure to use the correct prefix will result in a 'Data corruption detected' error. ```python # use VER_1 as prefix v1 = ZstdDict(VER_1, is_raw=True) # allow large window, the actual windowLog is from frame header. option = {DParameter.windowLogMax: 31} # get VER_2 from (VER_1 + PATCH) VER_2 = decompress(PATCH, zstd_dict=v1.as_prefix, option=option) ``` -------------------------------- ### ZstdCompressor compress() with Mode Parameter Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Shows how to use the compress() method with explicit mode arguments (FLUSH_BLOCK, FLUSH_FRAME) for more control over data flushing and frame management. ```python # use .compress() method with mode argument compressed_dat1 = c.compress(raw_dat1, c.FLUSH_BLOCK) compressed_dat2 = c.compress(raw_dat2, c.FLUSH_FRAME) ``` -------------------------------- ### hashLog Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Determines the size of the initial probe table, affecting compression ratio and speed. Special value 0 uses the default. ```APIDOC ## hashLog ### Description Size of the initial probe table, as a power of 2, resulting memory usage is ``1 << (hashLog+2)`` bytes. Must be clamped between ``ZSTD_HASHLOG_MIN`` and ``ZSTD_HASHLOG_MAX``. Larger tables improve compression ratio of strategies <= :py:attr:`~Strategy.dfast`, and improve speed of strategies > :py:attr:`~Strategy.dfast`. Special: value ``0`` means "use default hashLog", then the value is dynamically set, see "H" column in `this table `_. ``` -------------------------------- ### open() Function Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html The open() function provides a convenient way to open zstd-compressed files in either binary or text mode. It mirrors the functionality of bz2.open(), gzip.open(), and lzma.open() from the standard library. ```APIDOC ## open() ### Description Opens a zstd-compressed file in binary or text mode, returning a file-like object. Similar to standard library open functions for compression. ### Parameters - **filename** (str | bytes | path-like object | file object): The file to open or wrap. - **mode** (str): 'r', 'rb', 'w', 'wb', 'x', 'xb', 'a', 'ab' for binary mode; 'rt', 'wt', 'xt', 'at' for text mode. Default is 'rb'. - **level_or_option** (dict): Decompression options when in reading mode. Does not support integer compression levels. - **zstd_dict** (zstd_dict): A dictionary for Zstandard dictionary compression/decompression. - **encoding** (str, optional): The encoding to use for text mode. - **errors** (str, optional): Error handling scheme for text mode. - **newline** (str, optional): Controls universal newlines mode for text mode. ### Return Value - **ZstdFile**: In binary mode. - **io.TextIOWrapper**: In text mode, wrapping a ZstdFile object. ``` -------------------------------- ### Multi-threaded Compression Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Details how to enable multi-threaded compression using the `CParameter.nbWorkers` parameter. ```APIDOC ## Multi-threaded Compression ### Description Enable multi-threaded compression by setting the `CParameter.nbWorkers` parameter to a value greater than or equal to 1. ### Usage ```python # Enable multi-threaded compression with 4 workers option = {CParameter.nbWorkers : 4} compressed_dat = compress(raw_dat, option=option) ``` ### Parameters #### `CParameter.nbWorkers` - **Type**: Integer - **Description**: Set to `>= 1` to enable multi-threaded compression. `1` means "1-thread multi-threaded mode". The threads are spawned by the underlying zstd library, not by the pyzstd module. ``` -------------------------------- ### Loading and Using a Zstd Dictionary Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Load a zstd dictionary from a file and use it for compression. Reusing a compressor object is recommended for multiple compressions with the same dictionary for better performance. ```python # load a zstd dictionary from file with io.open(dict_path, 'rb') as f: file_content = f.read() zd = ZstdDict(file_content) # use the dictionary to compress. # if use a dictionary for compressor multiple times, reusing # a compressor object is faster, see .as_undigested_dict doc. compressed_dat = compress(raw_dat, zstd_dict=zd) ``` -------------------------------- ### Replace decompress_stream with pyzstd.open Source: https://pyzstd.readthedocs.io/en/latest/_sources/deprecated.md.txt Use `pyzstd.open` for stream decompression instead of `decompress_stream`. For more control, consider `EndlessZstdDecompressor`. ```python # before with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: decompress_stream(ifh, ofh) # after with pyzstd.open(input_file_path) as ifh: with io.open(output_file_path, 'wb') as ofh: shutil.copyfileobj(ifh, ofh) ``` ```python # after: more complex alternative with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: decompressor = EndlessZstdDecompressor() while True: if decompressor.needs_input: data = input_stream.read(read_size) if not data: break else: data = b"") ofh.write(decompressor.decompress(data, write_size)) callback_progress(ifh.tell(), ofh.tell()) # optional if not decompressor.at_frame_edge: raise ValueError("zstd data ends in an incomplete frame") ``` -------------------------------- ### Replace richmem_compress with compress Source: https://pyzstd.readthedocs.io/en/latest/deprecated.html Use the `pyzstd.compress` function for memory-efficient compression instead of `richmem_compress`. ```python # before data_out = pyzstd.richmem_compress(data_in, level_or_option=5) ``` ```python # after data_out = pyzstd.compress(data_in, level_or_option=5) ``` -------------------------------- ### ZstdCompressor Object Usage Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Demonstrates the basic usage of the ZstdCompressor object for compression and flushing data. ```APIDOC ## ZstdCompressor Object Usage ### Description This snippet shows how to instantiate a `ZstdCompressor` object with a specified compression option and then use it to compress raw data and flush any remaining compressed data. ### Code Example ```python c = ZstdCompressor(level_or_option=option) compressed_dat1 = c.compress(raw_dat) compressed_dat2 = c.flush() ``` ``` -------------------------------- ### Compressing with a Zstd Dictionary Prefix Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Compress data using a Zstd dictionary as a prefix. Ensure windowLog is clamped to the valid range and enable long distance matching for optimal results. The dictionary must be created with `is_raw=True`. ```python v1 = ZstdDict(VER_1, is_raw=True) # let the window cover the longest version. # don't forget to clamp windowLog to valid range. # enable "long distance matching". windowLog = max(len(VER_1), len(VER_2)).bit_length() option = {CParameter.windowLog: windowLog, CParameter.enableLongDistanceMatching: 1} # get a small PATCH PATCH = compress(VER_2, level_or_option=option, zstd_dict=v1.as_prefix) ``` -------------------------------- ### Replace decompress_stream with pyzstd.open Source: https://pyzstd.readthedocs.io/en/latest/deprecated.html Use `pyzstd.open` for stream decompression instead of `decompress_stream`. For more control, consider using `EndlessZstdDecompressor`. ```python # before with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: decompress_stream(ifh, ofh) ``` ```python # after with pyzstd.open(input_file_path) as ifh: with io.open(output_file_path, 'wb') as ofh: shutil.copyfileobj(ifh, ofh) ``` ```python # after: more complex alternative with io.open(input_file_path, 'rb') as ifh: with io.open(output_file_path, 'wb') as ofh: decompressor = EndlessZstdDecompressor() while True: if decompressor.needs_input: data = input_stream.read(read_size) if not data: break else: data = b"") ofh.write(decompressor.decompress(data, write_size)) callback_progress(ifh.tell(), ofh.tell()) # optional if not decompressor.at_frame_edge: raise ValueError("zstd data ends in an incomplete frame") ``` -------------------------------- ### strategy Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Selects the compression strategy, with higher values indicating more complex and slower compression. Special value 0 uses the default. ```APIDOC ## strategy ### Description See :py:attr:`Strategy` class definition. The higher the value of selected strategy, the more complex it is, resulting in stronger and slower compression. Special: value ``0`` means "use default strategy", then the value is dynamically set, see "strat" column in `this table `_. ``` -------------------------------- ### ZstdDict Initialization Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Initializes a ZstdDict object with dictionary content. The `is_raw` parameter allows for advanced usage with raw dictionary content. ```APIDOC ## ZstdDict.__init__ ### Description Initialize a ZstdDict object. ### Parameters * **dict_content** (_bytes-like object_) – Dictionary’s content. * **is_raw** (_bool_) – This parameter is for advanced user. `True` means _dict_content_ argument is a “raw content” dictionary, free of any format restriction. `False` means _dict_content_ argument is an ordinary zstd dictionary, was created by zstd functions, follow a specified format. ``` -------------------------------- ### Module-level variables Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Provides access to Zstandard library version information and compression level constants. ```APIDOC ## Module-level variables ### zstd_version Underlying zstd library's version as a string. #### Example ```python import pyzstd print(pyzstd.zstd_version) ``` ### zstd_version_info Underlying zstd library's version as a tuple. #### Example ```python import pyzstd print(pyzstd.zstd_version_info) ``` ### compressionLevel_values A namedtuple containing the minimum, maximum, and default compression levels supported by the underlying zstd library. #### Example ```python import pyzstd print(pyzstd.compressionLevel_values) ``` ### zstd_support_multithread A boolean indicating whether the underlying zstd library was compiled with multi-threaded compression support. #### Example ```python import pyzstd print(pyzstd.zstd_support_multithread) ``` ``` -------------------------------- ### Replace richmem_compress with compress Source: https://pyzstd.readthedocs.io/en/latest/_sources/deprecated.md.txt Use `pyzstd.compress` instead of `pyzstd.richmem_compress` for in-memory compression. ```python # before data_out = pyzstd.richmem_compress(data_in, level_or_option=5) # after data_out = pyzstd.compress(data_in, level_or_option=5) ``` -------------------------------- ### Compress with Digested vs. Undigested Dictionary Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Illustrates how to explicitly use a digested or undigested dictionary for compression. Undigested dictionaries are used by default and may offer better performance by avoiding a costly digestion operation. ```python compress(dat, zstd_dict=zd.as_digested_dict) ``` ```python compress(dat, zstd_dict=zd) ``` -------------------------------- ### chainLog Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Configures the size of the multi-probe search table, influencing compression quality and speed. Special value 0 uses the default. ```APIDOC ## chainLog ### Description Size of the multi-probe search table, as a power of 2, resulting memory usage is ``1 << (chainLog+2)`` bytes. Must be clamped between ``ZSTD_CHAINLOG_MIN`` and ``ZSTD_CHAINLOG_MAX``. Larger tables result in better and slower compression. This parameter is useless for :py:attr:`~Strategy.fast` strategy. It's still useful when using :py:attr:`~Strategy.dfast` strategy, in which case it defines a secondary probe table. Special: value ``0`` means "use default chainLog", then the value is dynamically set, see "C" column in `this table `_. ``` -------------------------------- ### compress() Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Compresses data using zstd. It can accept an integer for compression level or a dictionary for advanced parameters. A pre-trained dictionary can also be provided. ```APIDOC ## compress(data, level_or_option=None, zstd_dict=None) ### Description Compress _data_, return the compressed data. Compressing `b''` will get an empty content frame (9 bytes or more). ### Parameters * **data** (_bytes-like object_) – Data to be compressed. * **level_or_option** (_int_ _or_ _dict_) – When it’s an `int` object, it represents compression level. When it’s a `dict` object, it contains advanced compression parameters. The default value `None` means to use zstd’s default compression level/parameters. * **zstd_dict** (_ZstdDict_) – Pre-trained dictionary for compression. ### Returns Compressed data ### Return type bytes ### Request Example ```python # int compression level compressed_dat = compress(raw_dat, 10) # dict option, use 6 threads to compress, and append a 4-byte checksum. option = { CParameter.compressionLevel : 10, CParameter.nbWorkers : 6, CParameter.checksumFlag : 1 } compressed_dat = compress(raw_dat, option) ``` ``` -------------------------------- ### Compression Strategies (Strategy Enum) Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt Defines compression strategies from fastest to strongest, used for CParameter.strategy. ```APIDOC ## Strategy(IntEnum) ### Description Used for :py:attr:`~CParameter.strategy`. Compression strategies, listed from fastest to strongest. Note: new strategies might be added in the future, only the order (from fast to strong) is guaranteed. ### Members - `fast` - `dfast` - `greedy` - `lazy` - `lazy2` - `btlazy2` - `btopt` - `btultra` - `btultra2 ### Example ```python option = {CParameter.strategy : Strategy.lazy2, CParameter.checksumFlag : 1} compressed_dat = compress(raw_dat, option) ``` ``` -------------------------------- ### ZstdCompressor Methods Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt This section covers the methods available for the ZstdCompressor class, including compress, flush, and attributes related to compression modes. ```APIDOC ## ZstdCompressor.compress ### Description Provide data to the compressor object. ### Method compress(self, data, mode=ZstdCompressor.CONTINUE) ### Parameters #### Parameters - **data** (bytes-like object) - Data to be compressed. - **mode** (ZstdCompressor.CONTINUE, ZstdCompressor.FLUSH_BLOCK, ZstdCompressor.FLUSH_FRAME) - Compression mode. ### Returns - A chunk of compressed data if possible, or ``b''`` otherwise. ### Example ```python c = ZstdCompressor() compressed_data = c.compress(b'some data') ``` ## ZstdCompressor.flush ### Description Flush any remaining data in internal buffer. Since zstd data consists of one or more independent frames, the compressor object can still be used after this method is called. **Note**: Abuse of this method will reduce compression ratio, and some programs can only decompress single frame data. Use it only when necessary. ### Method flush(self, mode=ZstdCompressor.FLUSH_FRAME) ### Parameters #### Parameters - **mode** (ZstdCompressor.FLUSH_FRAME, ZstdCompressor.FLUSH_BLOCK) - Flush mode. ### Returns - Flushed data. ### Example ```python c = ZstdCompressor() data_to_flush = c.flush() ``` ## ZstdCompressor.last_mode ### Description The last mode used to this compressor, its value can be :py:attr:`~ZstdCompressor.CONTINUE`, :py:attr:`~ZstdCompressor.FLUSH_BLOCK`, :py:attr:`~ZstdCompressor.FLUSH_FRAME`. Initialized to :py:attr:`~ZstdCompressor.FLUSH_FRAME`. It can be used to get the current state of a compressor, such as, data flushed, a frame ended. ## ZstdCompressor.CONTINUE ### Description Used for *mode* parameter in :py:meth:`~ZstdCompressor.compress` method. Collect more data, encoder decides when to output compressed result, for optimal compression ratio. Usually used for traditional streaming compression. ## ZstdCompressor.FLUSH_BLOCK ### Description Used for *mode* parameter in :py:meth:`~ZstdCompressor.compress`, :py:meth:`~ZstdCompressor.flush` methods. Flush any remaining data, but don't close the current :ref:`frame`. Usually used for communication scenarios. If there is data, it creates at least one new :ref:`block`, that can be decoded immediately on reception. If no remaining data, no block is created, return ``b''``. **Note**: Abuse of this mode will reduce compression ratio. Use it only when necessary. ## ZstdCompressor.FLUSH_FRAME ### Description Used for *mode* parameter in :py:meth:`~ZstdCompressor.compress`, :py:meth:`~ZstdCompressor.flush` methods. Flush any remaining data, and close the current :ref:`frame`. Usually used for traditional flush. Since zstd data consists of one or more independent frames, data can still be provided after a frame is closed. **Note**: Abuse of this mode will reduce compression ratio, and some programs can only decompress single frame data. Use it only when necessary. ### Example ```python c = ZstdCompressor() compressed_dat1 = c.compress(raw_dat1, c.FLUSH_BLOCK) compressed_dat2 = c.compress(raw_dat2, c.FLUSH_FRAME) ``` ``` -------------------------------- ### Replace RichMemZstdCompressor with compress Source: https://pyzstd.readthedocs.io/en/latest/_sources/deprecated.md.txt Use `pyzstd.compress` for in-memory compression instead of `pyzstd.RichMemZstdCompressor`. ```python # before compressor = pyzstd.RichMemZstdCompressor(level_or_option=5) data_out1 = compressor.compress(data_in1) data_out2 = compressor.compress(data_in2) data_out3 = compressor.compress(data_in3) # after data_out1 = pyzstd.compress(data_in1, level_or_option=5) data_out2 = pyzstd.compress(data_in2, level_or_option=5) data_out3 = pyzstd.compress(data_in3, level_or_option=5) ``` -------------------------------- ### chainLog Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Configures the size of the multi-probe search table as a power of 2. Impacts compression quality and speed. ```APIDOC ## chainLog ### Description Sets the size of the multi-probe search table, expressed as a power of 2. Memory usage is `1 << (chainLog + 2)` bytes. Larger tables generally lead to better compression but slower performance. This parameter is not effective for the `fast` strategy. ### Parameter Details - **chainLog**: Size of the multi-probe search table, as a power of 2. Memory usage is `1 << (chainLog + 2)` bytes. Must be clamped between `ZSTD_CHAINLOG_MIN` and `ZSTD_CHAINLOG_MAX`. Larger tables result in better and slower compression. This parameter is useless for `fast` strategy. It’s still useful when using `dfast` strategy, in which case it defines a secondary probe table. Special value `0` means “use default chainLog”, then the value is dynamically set. ### Bounds Refer to `CParameter.chainLog.bounds()` for specific bounds. ``` -------------------------------- ### strategy Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Selects the compression strategy, influencing the trade-off between compression strength and speed. ```APIDOC ## strategy ### Description Chooses the compression strategy to use. Higher strategy values involve more complex algorithms, resulting in stronger compression but slower performance. A value of `0` indicates the default strategy, which is dynamically determined. ### Parameter Details - **strategy**: The higher the value of selected strategy, the more complex it is, resulting in stronger and slower compression. Special value `0` means “use default strategy”, then the value is dynamically set. ### Bounds Refer to `CParameter.strategy.bounds()` for specific bounds. ``` -------------------------------- ### compressionLevel Source: https://pyzstd.readthedocs.io/en/latest/pyzstd.html Sets compression parameters based on a pre-defined compression level. Manually set parameters are preserved. ```APIDOC ## compressionLevel ### Description Configures compression settings according to a predefined `compressionLevel` table. This affects dynamically adjustable parameters, while manually set parameters remain unchanged. ### Parameter Details - **compressionLevel**: Sets compression parameters according to a pre-defined table. See compression level for details. Setting a compression level does not set all other compression parameters to default. Setting this will dynamically impact the compression parameters which have not been manually set; the manually set ones will “stick”. ### Bounds Refer to `CParameter.compressionLevel.bounds()` for specific bounds. ``` -------------------------------- ### ZstdDecompressor Methods Source: https://pyzstd.readthedocs.io/en/latest/_sources/pyzstd.rst.txt This section details the ZstdDecompressor class for streaming decompression, including its initialization and decompression method. ```APIDOC ## ZstdDecompressor.__init__ ### Description Initialize a ZstdDecompressor object. ### Method __init__(self, zstd_dict=None, option=None) ### Parameters #### Parameters - **zstd_dict** (ZstdDict) - Pre-trained dictionary for decompression. - **option** (dict) - A ``dict`` object that contains :ref:`advanced decompression parameters`. The default value ``None`` means to use zstd's default decompression parameters. ## ZstdDecompressor.decompress ### Description Decompress *data*, returning decompressed data as a ``bytes`` object. ### Method decompress(self, data, max_length=-1) ### Parameters #### Parameters - **data** (bytes-like object) - Data to be decompressed. - **max_length** (int) - Maximum length of data to decompress. Defaults to -1 (no limit). ### Returns - Decompressed data as a ``bytes`` object. ### Example ```python d = ZstdDecompressor() decompressed_data = d.decompress(b'compressed data') ``` ```