### Install stream-zip with pip Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This command installs the latest version of the stream-zip library from PyPI using the pip package manager. It is the standard way to add the library to your Python environment. ```Shell pip install stream-zip ``` -------------------------------- ### Setting up and Previewing Documentation Locally - Bash Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Commands to install documentation dependencies (if not already installed) and start a local server to preview changes made to the documentation written with Material for mkdocs. Requires Python and pip. ```bash pip install -r requirements-docs.txt # Only needed once mkdocs serve ``` -------------------------------- ### Setup and serve mkdocs documentation locally - Bash Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Commands to install documentation dependencies (if needed) and serve the mkdocs documentation locally for previewing changes. ```Bash pip install -r requirements-docs.txt # Only needed once mkdocs serve ``` -------------------------------- ### Stream ZIP with generator yielding member files Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This example shows how to use a generator function to yield the member file definitions one by one. This pattern is useful for creating ZIP files from a large number of members without loading all definitions into memory simultaneously. ```Python from datetime import datetime from stat import S_IFREG from stream_zip import ZIP_32, stream_zip def member_files(): modified_at = datetime.now() mode = S_IFREG | 0o600 yield ('my-file-1.txt', modified_at, mode, ZIP_32, (b'Some bytes 1',)) yield ('my-file-2.txt', modified_at, mode, ZIP_32, (b'Some bytes 2',)) zipped_chunks = stream_zip(member_files()): for zipped_chunk in zipped_chunks: print(zipped_chunk) ``` -------------------------------- ### Setup Development Environment and Run Tests (Bash) Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Installs necessary dependencies and runs the existing test suite to ensure the development environment is correctly set up before making changes. ```Bash ./install-libarachive.sh # Only needed once pip install -r requirements-dev.txt # Only needed once pytest ``` -------------------------------- ### Setting Up Development Environment and Running Initial Tests Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Provides commands to install necessary dependencies and run the test suite for the first time to ensure the environment is set up correctly. Note that the install commands are typically only needed once. ```Bash ./install-libarachive.sh ``` ```Bash pip install -r requirements-dev.txt ``` ```Bash pytest ``` -------------------------------- ### Define a directory member file Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This snippet illustrates how to represent a directory within a ZIP file. The member name must end with a forward slash '/', the mode should include stat.S_IFDIR, and the content iterable must be empty. ```Python from datetime import datetime from stat import S_IFDIR from stream_zip import ZIP_32 directory = ('my-dir/', datetime.now(), S_IFDIR | 0o700, ZIP_32, ()) ``` -------------------------------- ### Basic stream-zip usage with tuple input Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This snippet demonstrates the basic usage of the stream_zip function by providing a tuple of member file definitions. Each member is a tuple containing the file name, modification time, mode, compression method, and an iterable of content chunks. ```Python from datetime import datetime from stat import S_IFREG from stream_zip import ZIP_32, stream_zip member_files = ( ( 'my-file-1.txt', # File name datetime.now(), # Modification time S_IFREG | 0o600, # Mode - regular file that owner can read and write ZIP_32, # ZIP_32 has good support but limited to 4GiB (b'Some bytes 1',), # Iterable of chunks of contents ), ( 'my-file-2.txt', datetime.now(), S_IFREG | 0o600, ZIP_32, (b'Some bytes 2',), ), ) zipped_chunks = stream_zip(member_files): for zipped_chunk in zipped_chunks: print(zipped_chunk) ``` -------------------------------- ### Define a symbolic link member file Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This code shows how to define a symbolic link within a ZIP file using stream-zip. The mode must include stat.S_IFLNK, and the content iterable should yield the bytes of the target path. ```Python from datetime import datetime from stat import S_IFLNK from stream_zip import ZIP_32 link = ('source.txt', datetime.now(), S_IFLNK | 0o600, ZIP_32, (b'target.txt',)) ``` -------------------------------- ### Stream ZIP with nested generators for members and content Source: https://github.com/uktrade/stream-zip/blob/main/docs/get-started.md This snippet demonstrates a more advanced streaming pattern where both the member file definitions and the content chunks for each file are provided by generator functions. This allows for efficient processing of large files and many members without excessive memory usage. ```Python from datetime import datetime from stat import S_IFREG from stream_zip import ZIP_32, stream_zip def member_files(): modified_at = datetime.now() mode = S_IFREG | 0o600 def file_1_data(): yield b'Some bytes 1' def file_2_data(): yield b'Some bytes 2' yield ('my-file-1.txt', modified_at, mode, ZIP_32, file_1_data()) yield ('my-file-2.txt', modified_at, mode, ZIP_32, file_2_data()) zipped_chunks = stream_zip(member_files()): for zipped_chunk in zipped_chunks: print(zipped_chunk) ``` -------------------------------- ### Compressing Local Files using stream-zip (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/input-examples.md This snippet demonstrates how to prepare data from named local files for compression using the stream-zip library. It defines a generator function that yields file metadata and content chunks for each specified file, suitable for passing to the stream_zip function. ```python from datetime import datetime from stat import S_IFREG from stream_zip import ZIP_32, stream_zip def local_files(names): now = datetime.now() def contents(name): with open(name, 'rb') as f: while chunk := f.read(65536): yield chunk return ( (name, now, S_IFREG | 0o600, ZIP_32, contents(name)) for name in names ) names = ('file-1.txt', 'file-2.txt') zipped_chunks = stream_zip(local_files(names)) ``` -------------------------------- ### Uploading Streamed ZIP to S3 with boto3 (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/output-examples.md This snippet demonstrates uploading a large, streamed ZIP file to Amazon S3 using `boto3`. It first converts the `stream_zip` output to a file-like object using `to_file_like_obj`, then uses `boto3.client('s3').upload_fileobj`. It includes configuration for `TransferConfig` to handle the unknown final size when streaming, setting a multipart chunk size. Requires `boto3`, `stream_zip`, and `to-file-like-obj` libraries, and an iterable `member_files`. ```python import boto3 from boto3.s3.transfer import TransferConfig from stream_zip import stream_zip from to_file_like_obj import to_file_like_obj zipped_chunks = stream_zip(member_files) zipped_chunks_obj = to_file_like_obj(zipped_chunks) s3 = boto3.client('s3') s3.upload_fileobj( zipped_chunks_obj, 'mybucket', 'mykey', # Since we're streaming the final total size is unknown, so we have to tell boto3 what part # size to use to accomodate the entire file - S3 has a hard coded limit of 10000 parts # In this example we choose a part size of 200MB, so 2TB maximum final object size Config=TransferConfig(multipart_chunksize=1024 * 1024 * 200), ) ``` -------------------------------- ### Converting stream-zip Output to File-like Object (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/output-examples.md This snippet shows how to convert the iterable output of `stream_zip` into a file-like object. It uses the `to_file_like_obj` function from the `to-file-like-obj` library to wrap the `zipped_chunks` iterable. This is useful when an API expects a file-like object instead of an iterable. Requires `stream_zip` and `to-file-like-obj` libraries and an iterable `member_files`. ```python from stream_zip import stream_zip from to_file_like_obj import to_file_like_obj zipped_chunks = stream_zip(member_files) zipped_chunks_obj = to_file_like_obj(zipped_chunks) ``` -------------------------------- ### Saving ZIP to Local File with stream-zip (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/output-examples.md This snippet demonstrates how to save the output of `stream_zip` to a local file using Python's built-in `open` function. It iterates through the `zipped_chunks` iterable and writes each chunk to a file named 'my.zip' in binary write mode. Requires the `stream_zip` library and an iterable `member_files`. ```python from stream_zip import stream_zip zipped_chunks = stream_zip(member_files) with open('my.zip', 'wb') as f: for chunk in zipped_chunks: f.write(chunk) ``` -------------------------------- ### Configure Zlib Compression Object (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/methods.md Example of a Python lambda function used for the `get_compressobj` parameter in `stream_unzip`. This specific configuration uses `level=0` to disable compression, resulting in the file being stored uncompressed despite using a 'compressed' method like `ZIP_32` or `ZIP_64`. ```Python lambda: zlib.compressobj(wbits=-zlib.MAX_WBITS, level=0) ``` -------------------------------- ### Running Tests After Making Changes Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Shows the command to run the test suite again after implementing code changes to verify functionality and prevent regressions. ```Bash pytest ``` -------------------------------- ### Creating a New Feature Branch Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Explains how to create a new local branch based on a descriptive name for your changes. ```Bash git checkout -b fix-a-bug-description ``` -------------------------------- ### Cloning Repository for uktrade Members - Bash Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Instructions for members of the uktrade GitHub organization to clone the stream-zip repository using SSH and navigate into the project directory. Requires an SSH key associated with the GitHub account. ```bash git clone git@github.com:uktrade/stream-zip.git cd stream-zup ``` -------------------------------- ### Cloning Forked Repository for Non-Members - Bash Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Instructions for individuals not members of the uktrade GitHub organization to clone their forked copy of the stream-zip repository using SSH and navigate into the project directory. Requires forking the repository and setting up an SSH key. ```bash git clone git@github.com:my-username/stream-zip.git cd stream-zup ``` -------------------------------- ### Committing Changes Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Details the commands to stage modified files and commit them to the local branch, ideally following the Conventional Commit specification for the message. ```Bash git add stream_zip.py ``` ```Bash git commit -m "feat: the bug description" ``` -------------------------------- ### Committing and Pushing Documentation Changes - Bash Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Git commands to stage modified documentation files, commit the changes with a descriptive message (ideally following Conventional Commits), and push the new branch to the remote fork. Prepares changes for a Pull Request. ```bash git add docs/recipies.md # Repeat for each file changed git commit -m "docs: added a Django recipe" gir push origin docs/add-django-recipe ``` -------------------------------- ### Creating Documentation Contribution Branch - Bash Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Command sequence to create a new Git branch specifically for documentation changes, prefixed with 'docs/', and then change the current directory to the project root. This isolates documentation work. ```bash git checkout -b docs/add-django-recipe cd stream-zip ``` -------------------------------- ### Pushing Changes to Remote Fork Source: https://github.com/uktrade/stream-zip/blob/main/docs/contributing.md Provides the command to push the local branch with committed changes to your remote fork on GitHub, making it available for creating a Pull Request. ```Bash gir push origin fix/the-bug-description ``` -------------------------------- ### Clone stream-zip for uktrade members - Bash Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Instructions for uktrade organisation members to clone the stream-zip repository using SSH after setting up an SSH key. ```Bash git clone git@github.com:uktrade/stream-zip.git cd stream-zup ``` -------------------------------- ### Create documentation branch - Bash Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Command to create a new Git branch specifically for documentation changes, following a descriptive naming convention. ```Bash git checkout -b docs/add-django-recipe cd stream-zip ``` -------------------------------- ### Clone forked stream-zip for non-members - Bash Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Instructions for users who are not members of the uktrade organisation to clone their forked stream-zip repository using SSH after setting up an SSH key and forking. ```Bash git clone git@github.com:my-username/stream-zip.git cd stream-zup ``` -------------------------------- ### Commit and push documentation changes - Bash Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Git commands to stage, commit, and push documentation changes to the remote repository, following conventional commit guidelines. ```Bash git add docs/recipies.md # Repeat for each file changed git commit -m "docs: added a Django recipe" gir push origin docs/add-django-recipe ``` -------------------------------- ### Stage, Commit, and Push Changes (Bash) Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Stages modified files for commit, creates a commit with a descriptive message following Conventional Commits, and pushes the changes to the contributor's fork. ```Bash git add stream_zip.py # Repeat for each file changed git commit -m "feat: the bug description" gir push origin fix/the-bug-description ``` -------------------------------- ### Generating Async Zip Stream with async_stream_zip Source: https://github.com/uktrade/stream-zip/blob/main/docs/async-interface.md Demonstrates how to use `async_stream_zip` to create a zip archive asynchronously. It shows how to define async iterables for file data and member file metadata, and how to iterate over the resulting async byte stream. Requires the `stream-zip` library and `asyncio`. ```python from datetime import datetime from stat import S_IFREG from stream_zip import async_stream_zip, ZIP_32 # Hard coded for example purposes async def async_data(): yield b'Some bytes 1' yield b'Some bytes 2' # Hard coded for example purposes async def async_member_files(): yield ( 'my-file-1.txt', datetime.now(), S_IFREG | 0o600, ZIP_32, async_data(), ) yield ( 'my-file-2.txt', datetime.now(), S_IFREG | 0o600, ZIP_32, async_data(), ) async def main(): async for chunk in async_stream_zip(async_member_files()): print(chunk) asyncio.run(main()) ``` -------------------------------- ### Run Tests After Making Changes (Bash) Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Executes the project's test suite after implementing code changes to verify that the modifications have not introduced regressions and new tests pass. ```Bash pytest ``` -------------------------------- ### Customizing Zlib Compression Options in Python Source: https://github.com/uktrade/stream-zip/blob/main/docs/advanced-usage.md Demonstrates how to override the default zlib compression object using the `get_compressobj` parameter to customize compression settings like `wbits` and `level`. Passing `level=0` disables compression. ```python for zipped_chunk in stream_zip(unzipped_files(), get_compressobj=lambda: zlib.compressobj(wbits=-zlib.MAX_WBITS, level=9)): print(zipped_chunk) ``` -------------------------------- ### Create New Feature Branch (Bash) Source: https://github.com/uktrade/stream-zip/blob/main/CONTRIBUTING.md Creates a new Git branch with a descriptive name prefixed by the change type, used for isolating changes during contribution. ```Bash git checkout -b fix-a-bug-description ``` -------------------------------- ### Encrypting ZIP Files with Password Protection in Python Source: https://github.com/uktrade/stream-zip/blob/main/docs/advanced-usage.md Explains how to encrypt ZIP file data using AES-256 by providing a `password` parameter. Adheres to the WinZip AE-2 specification and recommends using a strong, randomly generated password. ```python import secrets password = secrets.token_urlsafe(32) encrypted_zipped_chunks = stream_zip(member_files(), password=password) ``` -------------------------------- ### Using the ZIP_AUTO Method (Python) Source: https://github.com/uktrade/stream-zip/blob/main/docs/methods.md This snippet shows the signature for the ZIP_AUTO method, which dynamically selects the ZIP format (ZIP_32 or ZIP_64) based on the uncompressed size and other factors. It requires the uncompressed_size and optionally accepts a compression level. ```Python ZIP_AUTO(uncompressed_size, level=9) ``` -------------------------------- ### Setting Custom Chunk Size for Streaming ZIP in Python Source: https://github.com/uktrade/stream-zip/blob/main/docs/advanced-usage.md Shows how to set a custom `chunk_size` for processing data during streaming. This size is used for splitting/gathering both uncompressed input and compressed output data, potentially affecting performance. ```python for zipped_chunk in stream_zip(unzipped_files(), chunk_size=65536): print(zipped_chunk) ``` -------------------------------- ### Omitting Extended Timestamps in Python stream-zip Source: https://github.com/uktrade/stream-zip/blob/main/docs/advanced-usage.md Illustrates how to disable the inclusion of extended timestamps by setting `extended_timestamps=False`. This can reduce file size and is necessary for compatibility with formats like Open Document files. ```python for zipped_chunk in stream_zip(unzipped_files(), extended_timestamps=False): print(zipped_chunk) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.