### Install Smart Open with Optional Dependencies

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Installs smart_open along with specified optional dependencies for various storage solutions. Use '[all]' to install all available dependencies.

```sh
pip install 'smart_open[s3,gcs,azure,http,webhdfs,ssh,zst,lz4]'
```

```sh
pip install 'smart_open[all]'
```

--------------------------------

### Install All Dependencies in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

To maintain backward compatibility with versions prior to 3.0.0, you can install all dependencies by using the '[all]' extra. This ensures all storage backends are available.

```bash
pip install smart_open[all]
```

--------------------------------

### Install Specific Dependencies in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Version 3.0.0 introduced a new installation method where dependencies are not installed by default. Use extras like '[s3]' to install only the required dependencies for specific storage backends.

```bash
pip install smart_open[s3]
```

--------------------------------

### Install Dependencies

Source: https://github.com/piskvorky/smart_open/blob/develop/integration-tests/README.md

Install the necessary dependencies, including py.test and its benchmarks addon, using the provided requirements file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install HTTP Support in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Starting from version 4.0.1, the 'requests' library is not installed by default. Install the 'http' extra to enable support for http, https, and webhdfs URLs.

```diff
- pip install smart_open
+ pip install smart_open[http]
```

--------------------------------

### Activate Virtual Environment and Install Dependencies

Source: https://github.com/piskvorky/smart_open/blob/develop/CONTRIBUTING.md

Activate the created virtual environment and install development dependencies along with pre-commit hooks. This prepares the environment for development and testing.

```sh
. .venv/bin/activate
make install
```

--------------------------------

### Clone Repository and Set Up Virtual Environment

Source: https://github.com/piskvorky/smart_open/blob/develop/CONTRIBUTING.md

Clone the smart_open repository and create a Python virtual environment. This is the initial setup step for development.

```sh
git clone git@github.com:RaRe-Technologies/smart_open.git
cd smart_open
python -m venv .venv
```

--------------------------------

### Install Test Dependencies

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Install the necessary development dependencies for running tests. This command should be run from the project's root directory.

```sh
pip install -e .[dev]
```

--------------------------------

### Python Doctest Example

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

This snippet demonstrates a basic Python doctest example. Lines starting with '>>>' are commands, and subsequent lines are expected output. Ensure examples are enclosed in triple backticks and end with a blank line.

```python
>>> foo = 'bar'
>>> print(foo)
bar

```

--------------------------------

### Read Zip Files with Smart Open and Zipfile

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Example of reading contents from a zip file using smart_open for I/O and zipfile for compression/decompression. Requires 'sampledata/hello.zip' to exist.

```python
>>> from smart_open import open
>>> import zipfile
>>> with open('sampledata/hello.zip', 'rb') as fin:
...     with zipfile.ZipFile(fin) as zip:
...         for info in zip.infolist():
...             file_bytes = zip.read(info.filename)
...             print('%r: %r' % (info.filename, file_bytes.decode('utf-8')))
'hello/': ''
'hello/en.txt': 'hello world!\n'
'hello/ru.txt': 'здравствуй, мир!\n'

```

--------------------------------

### Register Custom Compressor for .xz Files

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Demonstrates how to register a custom compressor for a new file extension. This example shows registering support for .xz files using the `lzma` module.

```python
>>> import lzma, os
>>> from smart_open import open, register_compressor

>>> def _handle_xz(file_obj, mode, **kwargs):
...      return lzma.open(filename=file_obj, mode=mode, **kwargs)

>>> register_compressor('.xz', _handle_xz)

>>> with open('tests/test_data/1984.txt.xz') as fin:
...     print(fin.read(32))
It was a bright cold day in Apri

```

--------------------------------

### Write Zip Files with Smart Open and Zipfile

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Example of writing data to a zip file using smart_open for I/O and zipfile for compression/decompression. A temporary file is created and then deleted.

```python
>>> from smart_open import open
>>> import os
>>> import tempfile
>>> import zipfile
>>> tmp = tempfile.NamedTemporaryFile(prefix='smart_open-howto-', suffix='.zip', delete=False)
>>> with open(tmp.name, 'wb') as fout:
...    with zipfile.ZipFile(fout, 'w') as zip:
...       zip.writestr('hello/en.txt', 'hello world!\n')
...       zip.writestr('hello/ru.txt', 'здравствуй, мир!\n')
>>> os.unlink(tmp.name)  # comment this line to keep the file for later

```

--------------------------------

### Run Unit Tests

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Execute the unit test suite using pytest. Ensure you have installed the test dependencies first.

```sh
pytest tests
```

--------------------------------

### Download Directory from Google Cloud Storage

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Iterate through blobs in a GCS bucket with a specified prefix to download directory contents. This example shows how to list and access files within a 'directory' on GCS.

```python
>>> from google.cloud import storage
>>> from smart_open import open
>>> client = storage.Client()
>>> bucket_name = "gcp-public-data-landsat"
>>> prefix = "LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/"
>>> for blob in client.list_blobs(client.get_bucket(bucket_name), prefix=prefix):
...      with open(f"gcs://{bucket_name}/{blob.name}") as f:
...          print(f.name)
...          break # just show the first iteration for the test
LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_ANG.txt

```

--------------------------------

### Read/Write to Localstack S3 Bucket

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Interact with a localstack S3 bucket for testing purposes. Requires localstack to be installed and running, and a bucket to be created.

```python
import boto3
from smart_open import open
client = boto3.client('s3', endpoint_url='http://localhost:4566')
tparams = {'client': client}
with open('s3://mybucket/hello.txt', 'wt', transport_params=tparams) as fout:
    _ = fout.write('hello world!')
with open('s3://mybucket/hello.txt', 'rt', transport_params=tparams) as fin:
    fin.read()
```

--------------------------------

### Write Compressed File with Custom Compression Level

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes to a compressed file, passing custom options to the compression library via `compression_kwargs`. This example lowers the gzip compresslevel for faster writes.

```python
>>> import tempfile
>>> from smart_open import open
>>> with tempfile.NamedTemporaryFile(suffix='.gz') as tmp:
...     with open(tmp.name, 'wb', compression_kwargs={'compresslevel': 6}) as fout:
...         _ = fout.write(b'hello world')


```

--------------------------------

### Access S3 Object with Custom Client

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Opens an object from AWS S3 using `smart_open`, providing a pre-configured `boto3` client via `transport_params`. Ensure `boto3` is installed and configured.

```python
>>> import boto3
>>> fin = open('s3://commoncrawl/robots.txt', transport_params=dict(client=boto3.client('s3')))

```

--------------------------------

### New Transport Mechanism Structure

Source: https://github.com/piskvorky/smart_open/blob/develop/EXTENDING.md

Defines the essential components required for a new transport mechanism module in smart_open. This includes schema definition, URI examples, handling missing dependencies, parsing URIs, and opening URIs with transport-specific parameters.

```python
SCHEMA = ...
"""The name of the mechanism, e.g. s3, ssh, etc.

This is the part that goes before the `://` in a URL, e.g. `s3://`."""

URI_EXAMPLES = ('xxx://foo/bar', 'zzz://baz/boz')
"""This will appear in the documentation of the the `parse_uri` function."""

MISSING_DEPS = False
"""Wrap transport-specific imports in a try/catch and set this to True if
any imports are not found. Seting MISSING_DEPS to True will cause the library
to suggest installing its dependencies with an example pip command.

If your transport has no external dependencies, you can omit this variable.
"""

def parse_uri(uri_as_str):
    """Parse the specified URI into a dict.

    At a bare minimum, the dict must have `schema` member.
    """
    return dict(schema=XXX_SCHEMA, ...)


def open_uri(uri_as_str, mode, transport_params):
    """Return a file-like object pointing to the URI.

    Parameters:

    uri_as_str: str
        The URI to open
    mode: str
        Either "rb" or "wb".  You don't need to implement text modes,
        `smart_open` does that for you, outside of the transport layer.
    transport_params: dict
        Any additional parameters to pass to the `open` function (see below).

    """
    #
    # Parse the URI using parse_uri
    # Consolidate the parsed URI with transport_params, if needed
    # Pass everything to the open function (see below).
    #
    ...


def open(..., mode, param1=None, param2=None, paramN=None):
    """This function does the hard work.

    The keyword parameters are the transport_params from the `open_uri`
    function.

    """
    ...
```

--------------------------------

### Perform Single-Part Upload to S3

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes bytes to an S3 object using a single-part upload, which can save API requests and allows seeking before upload. This example uses the default io.BytesIO buffer.

```python
# perform a single-part upload to S3 (saves billable API requests, and allows seek() before upload)
with open(
    "s3://smart-open-py37-benchmark-results/test.txt", "wb", transport_params={"multipart_upload": False}
) as fout:
    bytes_written = fout.write(b"hello world!")
    print(bytes_written)
```

--------------------------------

### Display Smart Open Open Function Help

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Prints the help documentation for the `smart_open.open` function, which details supported URI schemes and transport-specific parameters.

```python
help("smart_open.open")

```

--------------------------------

### Use smart_open.open directly

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The compatibility wrapper `smart_open.smart_open()` has been removed. Call `smart_open.open()` directly. If `ignore_extension=True` was used, switch to `compression='disable'`.

```diff
fin = smart_open.smart_open('s3://bucket/key.gz', 'rb', ignore_extension=True)
fin = smart_open.open('s3://bucket/key.gz', 'rb', compression='disable')
```

--------------------------------

### Run Release Script

Source: https://github.com/piskvorky/smart_open/blob/develop/release/README.md

Execute the release script to update the changelog and prepare the release branches.

```bash
bash release/release.sh
```

--------------------------------

### Import from smart_open.utils

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Underscored re-export aliases in `smart_open.smart_open_lib` are gone. Import canonical names from `smart_open.utils`. The public `smart_open.register_compressor` remains functional.

```diff
from smart_open.smart_open_lib import _check_kwargs, _inspect_kwargs
from smart_open.utils import check_kwargs, inspect_kwargs
```

--------------------------------

### Registering a New Transport

Source: https://github.com/piskvorky/smart_open/blob/develop/EXTENDING.md

Demonstrates how to register a newly implemented transport module within the smart_open.transport submodule. This ensures the new transport is recognized and usable by the smart_open library.

```python
python -c 'help("smart_open")'
```

--------------------------------

### Stream from/to Compressed Files

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Shows how to read from and write to compressed files (e.g., .gz, .bz2) with transparent decompression/compression. Ensure the file is specified with the correct extension.

```python
>>> # stream from/to compressed files, with transparent (de)compression:
>>> for line in open('tests/test_data/1984.txt.gz', encoding='utf-8'):
...    print(repr(line))
'It was a bright cold day in April, and the clocks were striking thirteen.\n'
'Winston Smith, his chin nuzzled into his breast in an effort to escape the vile\n'
'wind, slipped quickly through the glass doors of Victory Mansions, though not\n'
'quickly enough to prevent a swirl of gritty dust from entering along with him.\n'
```

```python
>>> # can use context managers too:
>>> with open('tests/test_data/1984.txt.gz') as fin:
...    with open('tests/test_data/1984.txt.bz2', 'w') as fout:
...        for line in fin:
...           fout.write(line)
74
80
78
79
```

--------------------------------

### Import iter_bucket from smart_open.s3

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The top-level `smart_open.s3_iter_bucket` wrapper has been removed. Import `iter_bucket` from `smart_open.s3` instead.

```diff
from smart_open import s3_iter_bucket
from smart_open.s3 import iter_bucket as s3_iter_bucket
```

--------------------------------

### View All Make Targets

Source: https://github.com/piskvorky/smart_open/blob/develop/CONTRIBUTING.md

Display all available targets for the make build system. This is useful for discovering available commands and tasks.

```sh
make help
```

--------------------------------

### Register New Compressor with smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/EXTENDING.md

Use this pattern to add support for a new compression format. Create a handler function that takes a file object and mode, then register it with the desired file extension.

```python
def _handle_xz(file_obj, mode):
    import lzma

    return lzma.LZMAFile(filename=file_obj, mode=mode)


register_compressor(".xz", _handle_xz)
```

--------------------------------

### Register Custom Compressor for .xz Files

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Demonstrates how to register a custom compressor callback for files with a specific extension. The callback should accept file object, mode, and keyword arguments for flexibility.

```python
def _handle_xz(file_obj, mode, **kwargs):
    import lzma

    return lzma.open(filename=file_obj, mode=mode, **kwargs)

register_compressor(".xz", _handle_xz)
```

--------------------------------

### View Smart Open API Reference

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Accesses the smart_open API reference using Python's built-in help function.

```python
help("smart_open")
```

--------------------------------

### Update Imports: from smart_open import smart_open to from smart_open import open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Replace the import statement for `smart_open` with the new `open` function.

```diff
from smart_open import smart_open
+ from smart_open import open
```

--------------------------------

### Run S3 Performance Benchmarks

Source: https://github.com/piskvorky/smart_open/blob/develop/integration-tests/README.md

Initiate performance benchmarks for S3 operations using SMART_OPEN_S3_URL to specify the bucket and test path. This command runs the tests and collects performance statistics.

```bash
$ SMART_OPEN_S3_URL=s3://bucket/smart_open_test py.test integration-tests/test_s3.py
```

--------------------------------

### Configure GCS Credentials with google.cloud.storage.Client

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Pass a pre-configured google.cloud.storage.Client object to smart_open to manage GCS authentication. Ensure GOOGLE_APPLICATION_CREDENTIALS environment variable is set.

```python
import os
from google.cloud.storage import Client

service_account_path = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
client = Client.from_service_account_json(service_account_path)
fin = open("gcs://gcp-public-data-landsat/index.csv.gz", transport_params=dict(client=client))
```

--------------------------------

### Remove compression.tweak_close() call

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The `smart_open.compression.tweak_close()` helper has been removed as its functionality is now handled by the standard `__exit__` method.

--------------------------------

### Remove GCS buffer_size and line_terminator parameters

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The `buffer_size` and `line_terminator` parameters for `smart_open.gcs.open()` and `smart_open.gcs.Reader()` have been removed. These parameters were no-ops and emitted warnings.

```diff
- smart_open.gcs.open(bucket, blob, 'rb', buffer_size=8192)
+ smart_open.gcs.open(bucket, blob, 'rb')
```

--------------------------------

### Set Object ACL Directly with boto3

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Demonstrates setting an object's ACL directly using boto3 after the object has been created. This approach bypasses smart_open for ACL manipulation.

```python
import boto3
import smart_open

with open("s3://bucket/key", "wb") as fout:
    fout.write(b"hello world!")
client = boto3.client("s3")
client.put_object_acl(ACL=acl_as_string)

```

--------------------------------

### Run S3 Integration Tests

Source: https://github.com/piskvorky/smart_open/blob/develop/integration-tests/README.md

Execute the S3 integration tests by setting the SO_BUCKET and SO_KEY environment variables and running py.test on the specific test file.

```bash
SO_BUCKET=bucket SO_KEY=key py.test integration-tests/test_s3.py
```

--------------------------------

### Run Linting Tools

Source: https://github.com/piskvorky/smart_open/blob/develop/CONTRIBUTING.md

Execute linting tools such as ruff, pydoclint, and yamlfmt. This ensures code quality and adherence to project standards, mirroring the CI checks.

```sh
make lint
```

--------------------------------

### Context Manager for File Copying

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Use context managers to read from one compressed file and write to another, performing transparent compression/decompression.

```python
>>> # can use context managers too:
>>> with open('tests/test_data/1984.txt.gz') as fin:
...    with open('tests/test_data/1984.txt.bz2', 'w') as fout:
...        for line in fin:
...           fout.write(line)
74
80
78
79
```

--------------------------------

### Explicitly Specify Binary Mode 'rb'

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

If your code relied on the default 'rb' mode of the old `smart_open.smart_open`, explicitly pass 'rb' to the new `smart_open.open` function.

```diff
smart_open.smart_open('s3://commoncrawl/robots.txt').read(32)  # 'rb' used to be the default
+ smart_open.open('s3://commoncrawl/robots.txt', 'rb').read(32)
```

--------------------------------

### Disable Compression in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Use the `compression` parameter set to 'disable' to explicitly turn off compression, replacing the deprecated `ignore_ext` parameter.

```diff
- fin = smart_open.open("/path/file.gz", ignore_ext=True)
+ fin = smart_open.open("/path/file.gz", compression="disable")
```

```diff
- fin = smart_open.open("/path/file.gz", ignore_ext=False)
+ fin = smart_open.open("/path/file.gz")
```

--------------------------------

### smart_open.open

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Opens a specified URI, returning a file-like object for reading or writing. It supports various URI formats, pathlib.Path objects, and streams, with options for compression and transport layer parameters.

```APIDOC
## smart_open.open

### Description
Opens the given file for reading/writing. Supports S3, HDFS, local filesystem, compressed files, and more using a simple, Pythonic API. The streaming makes heavy use of generators and pipes to avoid loading full file contents into memory, allowing work with arbitrarily large files.

### Method
open

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **uri** (Uri) - The object to open. Can be a string, pathlib.Path instance, or a stream.
* **mode** (str, optional) - Mimics built-in open parameter. Defaults to 'r'.
* **buffering** (int, optional) - Mimics built-in open parameter. Defaults to -1.
* **encoding** (str | None, optional) - Mimics built-in open parameter.
* **errors** (str | None, optional) - Mimics built-in open parameter.
* **newline** (str | None, optional) - Mimics built-in open parameter.
* **closefd** (bool, optional) - Mimics built-in open parameter. Ignored. Defaults to True.
* **opener** (Callable[[str, int], int] | None, optional) - Mimics built-in open parameter. Ignored.
* **compression** (str, optional) - Explicitly specify the compression/decompression behavior. Defaults to 'infer_from_extension'.
* **compression_kwargs** (CompressionKwargs | None, optional) - Keyword arguments forwarded to the registered compressor callback.
* **transport_params** (TransportParams | None, optional) - Additional parameters for the transport layer.

### Returns
A file-like object.

### Raises
* **TypeError**: If ``mode`` is not a string or if the URI type is not recognized.
* **ValueError**: If ``compression`` is not a supported value.
* **NotImplementedError**: If ``mode`` cannot be parsed into a valid binary mode.

### Notes
- smart_open has several implementations for its transport layer (e.g. S3, HTTP). Each transport layer has a different set of keyword arguments for overriding default behavior. If you specify a keyword argument that is *not* supported by the transport layer being used, smart_open will ignore that argument and log a warning message.
- Transports include azure, ftp, gcs, hdfs, http, local_file, s3, ssh, webhdfs.
```

--------------------------------

### Pre-configure boto3 Client with Standard Retries for S3

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Pass a pre-configured boto3 client with standard retry mode to smart_open for robust S3 operations. This ensures that recoverable network errors are automatically retried.

```python
import boto3
import botocore.config
import smart_open
config = botocore.config.Config(retries={'mode': 'standard'})
client = boto3.client('s3', config=config)
tp = {'client': client}
with open('s3://commoncrawl/robots.txt', transport_params=tp) as fin:
    print(fin.readline())

```

--------------------------------

### Infer or Specify Compression in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The `compression` parameter can be used to infer compression from file extensions or explicitly set a compression codec.

```python
fin = smart_open.open(
    "/path/file.gz", compression="infer_from_extension"  # the default; explicit if you prefer
)
fin = smart_open.open("/path/file", compression=".gz")
```

--------------------------------

### IOBase Operations with Seek

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Demonstrates using IOBase operations like 'seek' on a file opened from S3. The file is opened in binary mode ('rb').

```python
>>> # can use any IOBase operations, like seek
>>> with open('s3://commoncrawl/robots.txt', 'rb') as fin:
...     for line in fin:
...         print(repr(line.decode('utf-8')))
...         break
...     offset = fin.seek(0)  # seek to the beginning
...     print(fin.read(4))
'User-Agent: *\n'
b'User'
```

--------------------------------

### Run Test Suite

Source: https://github.com/piskvorky/smart_open/blob/develop/CONTRIBUTING.md

Execute the project's test suite to ensure code integrity and functionality. This command runs all automated tests.

```sh
make test
```

--------------------------------

### Read File from Github API

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Use smart_open to read a file directly from the Github API. Requires a GITHUB_TOKEN environment variable for authentication.

```python
import base64
import gzip
import json
import os
from smart_open import open
owner, repo, path = "RaRe-Technologies", "smart_open", "HOWTO.md"
github_token = os.environ['GITHUB_TOKEN']
url = f"https://api.github.com/repos/{owner}/{repo}/contents/{path}"
params = {"headers" : {"Authorization" : "Bearer " + github_token}}
with open(url, 'rb', transport_params=params) as fin:
    response = json.loads(gzip.decompress(fin.read()))
response["path"]
```

--------------------------------

### Migrate S3 Session to Client in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

When migrating from older smart_open versions that used a boto3 session, construct an S3 client from the session and pass it via `transport_params`.

```diff
- smart_open.open('s3://bucket/key', transport_params={'session': session})
+ smart_open.open('s3://bucket/key', transport_params={'client': session.client('s3')})
```

--------------------------------

### Replace create_pool with ThreadPoolExecutor

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Use `smart_open.concurrency.ThreadPoolExecutor` directly instead of the removed `create_pool` context manager. Note the change from `imap_unordered` to `imap`.

```diff
- with smart_open.concurrency.create_pool(processes=8) as pool:
-     for result in pool.imap_unordered(fn, items):
-         ...
+ with smart_open.concurrency.ThreadPoolExecutor(max_workers=8) as pool:
+     for result in pool.imap(fn, items):
+         ...
```

--------------------------------

### Migrate Transport Parameters using transport_params

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Bundle transport-specific keyword parameters into a `transport_params` dictionary when using the new `smart_open.open` function.

```diff
session = boto3.Session(profile_name='smart_open')
- smart_open.smart_open(url, 'r', session=session).read(32)
+ params = {'session': boto3.Session(profile_name='smart_open')}
+ open(url, 'r', transport_params=params).read(32)
```

--------------------------------

### Read File with Explicit Compression Algorithm

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Specifies the compression algorithm (e.g., .gz) explicitly, useful for non-standard file extensions. Ensure the file is compressed with the specified algorithm.

```python
>>> from smart_open import open
>>> with open('tests/test_data/1984.txt.gzip', compression='.gz') as fin:
...     print(fin.read(32))
It was a bright cold day in Apri

```

--------------------------------

### Configure GCS Credentials with google.auth.credentials.Credentials

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Create an explicit google.auth.credentials.Credentials object with an API token and pass it to the google.cloud.storage.Client for GCS authentication.

```python
import os
from google.auth.credentials import Credentials
from google.cloud.storage import Client

token = os.environ["GOOGLE_API_TOKEN"]
credentials = Credentials(token=token)
client = Client(credentials=credentials)
fin = open("gcs://gcp-public-data-landsat/index.csv.gz", transport_params={"client": client})
```

--------------------------------

### Efficient S3 Reading with Shared Boto3 Client

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Use this pattern to share a boto3 S3 client across multiple smart_open calls for improved efficiency. Configure the client with specific retry and connection pool settings.

```python
import boto3
import botocore.client
from smart_open import open
# this mirrors the adaptive-retry config smart_open uses internally when it
# creates a thread-safe client for you (see smart_open.s3)
config = botocore.client.Config(
    max_pool_connections=64,
    tcp_keepalive=True,
    retries={'max_attempts': 6, 'mode': 'adaptive'},
)
transport_params = {'client': boto3.client('s3', config=config)}
for month in (1, 2, 3):
    url = 's3://nyc-tlc/trip data/yellow_tripdata_2020-%02d.csv' % month
    with open(url, transport_params=transport_params) as fin:
        _ = fin.readline()  # skip CSV header
        print(fin.readline().strip())

```

--------------------------------

### Patch pathlib.Path.open for Compressed Files

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Replace the built-in Path.open with smart_open.open to enable reading compressed files. This is useful when working with formats like .gz.

```python
>>> from pathlib import Path
>>> from smart_open.smart_open_lib import patch_pathlib
>>>
>>> _ = patch_pathlib()  # replace `Path.open` with `smart_open.open`
>>>
>>> path = Path("tests/test_data/crime-and-punishment.txt.gz")
>>>
>>> with path.open("r") as infile:
...     print(infile.readline()[:41])
В начале июля, в чрезвычайно жаркое время

```

--------------------------------

### Migrate S3 Resource to Client in smart_open

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Replace the use of a boto3 S3 resource with an S3 client when passing it via `transport_params` in smart_open.

```diff
- resource = session.resource('s3', **resource_kwargs)
- smart_open.open('s3://bucket/key', transport_params={'resource': resource})
+ client = session.client('s3')
+ smart_open.open('s3://bucket/key', transport_params={'client': client})
```

--------------------------------

### register_compressor

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Registers a callback function for transparently decompressing files with a specific extension. This allows smart_open to handle custom compression formats.

```APIDOC
## register_compressor

### Description
Registers a callback for transparently decompressing files with a specific extension.

### Method
`register_compressor(ext: 'str', callback: 'Compressor') -> 'None'`

### Parameters
#### Arguments
- **ext** (str) - Required - The extension of the file to be decompressed. Must include the leading period, e.g. `.gz`.
- **callback** (Compressor) - Required - The callback function. It must accept two positional arguments, `file_obj` and `mode`, and should ideally accept `**kwargs` to handle additional compression arguments passed via `smart_open.open(..., compression_kwargs={...})`.

### Raises
- **ValueError**: If `ext` does not start with a period.

### Example
```python
def _handle_xz(file_obj, mode, **kwargs):
    import lzma

    return lzma.open(filename=file_obj, mode=mode, **kwargs)

register_compressor(".xz", _handle_xz)
```
```

--------------------------------

### Read Compressed File with Default Compression

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads a .gz compressed file using the default inference behavior. Ensure the file exists at the specified path.

```python
>>> from smart_open import open
>>> with open('tests/test_data/1984.txt.gz') as fin:
...     print(fin.read(32))
It was a bright cold day in Apri

```

--------------------------------

### Enable Debug Logging for smart_open S3

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Enable debug logging for the smart_open.s3 module to verify retry settings and diagnose issues. Check the log output after running your S3 operations.

```python
import logging

logging.getLogger("smart_open.s3").setLevel(logging.DEBUG)

```

--------------------------------

### Pass session_kwargs as a dict to s3.iter_bucket

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

`smart_open.s3.iter_bucket` no longer accepts session keyword arguments via `**session_kwargs`. Pass a single `session_kwargs` dictionary instead.

```diff
smart_open.s3.iter_bucket(
    bucket,
    aws_access_key_id='id',
    aws_secret_access_key='secret',
    session_kwargs={
        'aws_access_key_id': 'id',
        'aws_secret_access_key': 'secret',
    },
)
```

--------------------------------

### Iterate Over S3 Bucket Contents in Parallel

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Use smart_open.s3.iter_bucket to efficiently process bucket keys in parallel using multithreading. Filter keys with 'accept_key' and limit results with 'key_limit'.

```python
>>> from smart_open import s3
>>> # we use workers=1 for reproducibility; you should use as many workers as you have cores
>>> bucket = 'silo-open-data'
>>> prefix = 'Official/annual/monthly_rain/'
>>> for key, content in s3.iter_bucket(bucket, prefix=prefix, accept_key=lambda key: '/201' in key, workers=1, key_limit=3):
...     print(key, round(len(content) / 2**20))
Official/annual/monthly_rain/2010.monthly_rain.nc 13
Official/annual/monthly_rain/2011.monthly_rain.nc 13
Official/annual/monthly_rain/2012.monthly_rain.nc 13

```

--------------------------------

### Configure S3 Credentials with boto3 Session

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Pass a customized boto3.Session object to smart_open to control S3 credentials. This method is mutually exclusive with credentials in the S3 URL.

```python
session = boto3.Session(
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)
client = session.client("s3", endpoint_url=..., config=...)
fin = open("s3://bucket/key", transport_params={"client": client})
```

--------------------------------

### Set Object ACL via smart_open Transport Parameters

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Pass the 'ACL' parameter for 'S3.Client.create_multipart_upload' via smart_open's transport_params to set the object's ACL during creation. This is an alternative to direct boto3 manipulation.

```python
import smart_open

tp = {"client_kwargs": {"S3.Client.create_multipart_upload": {"ACL": acl_as_string}}}
with open("s3://bucket/key", "wb", transport_params=tp) as fout:
    fout.write(b"hello world!")

```

--------------------------------

### Update GCS URI Scheme to gcs://

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Prefer 'gcs://' over 'gs://' for new GCS URIs. The 'gs://' scheme remains functional as an alias for backward compatibility.

```diff
- smart_open.open('gs://my_bucket/my_file.txt')
+ smart_open.open('gcs://my_bucket/my_file.txt')
```

--------------------------------

### Pass Additional Parameters to boto3 S3 get_object

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Pass additional parameters, such as 'RequestPayer', to specific boto3 client functions like 'S3.Client.get_object' via smart_open's transport_params. This influences how smart_open calls the underlying boto3 function.

```python
import boto3
from smart_open import open
transport_params = {'client_kwargs': {'S3.Client.get_object': {'RequestPayer': 'requester'}}}
with open('s3://arxiv/pdf/arXiv_pdf_manifest.xml', transport_params=params) as fin:
   print(fin.readline())

```

--------------------------------

### Propagate boto3 Client Arguments for S3 Uploads

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Use the 'client_kwargs' transport parameter to pass arguments like Metadata, ACL, and StorageClass to boto3's S3 client methods for advanced upload configurations.

```python
kwargs = {"Metadata": {"version": 2}, "ACL": "authenticated-read", "StorageClass": "STANDARD_IA"}
fout = open(
    "s3://bucket/key", "wb", transport_params={"client_kwargs": {"S3.Client.create_multipart_upload": kwargs}}
)
```

--------------------------------

### Stream from Compressed Files

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Read from a gzipped file with transparent decompression. The 'encoding' parameter is used for text mode.

```python
>>> # stream from/to compressed files, with transparent (de)compression:
>>> for line in open('tests/test_data/1984.txt.gz', encoding='utf-8'):
...    print(repr(line))
'It was a bright cold day in April, and the clocks were striking thirteen.\n'
'Winston Smith, his chin nuzzled into his breast in an effort to escape the vile\n'
'wind, slipped quickly through the glass doors of Victory Mansions, though not\n'
'quickly enough to prevent a swirl of gritty dust from entering along with him.\n'
```

--------------------------------

### Migrating S3 Endpoint Configuration

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

Illustrates the change in how custom S3 endpoint URLs are specified. Previously, `resource_kwargs` was used; now, a `boto3` client configured with the endpoint URL is passed directly via `transport_params`.

```diff
  session = boto3.Session(profile_name='digitalocean')
- resource_kwargs = {'endpoint_url': 'https://ams3.digitaloceanspaces.com'}
- with open('s3://bucket/key.txt', 'wb', transport_params={'resource_kwarg': resource_kwargs}) as fout:
+ client = session.client('s3', endpoint_url='https://ams3.digitaloceanspaces.com')
+ with open('s3://bucket/key.txt', 'wb', transport_params={'client': client}) as fout:
      fout.write(b'here we stand')
```

--------------------------------

### Stream from HTTP

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Read data from a remote HTTP resource. This snippet shows how to stream content from a URL.

```python
>>> # stream from HTTP
>>> for line in open('http://example.com'):
...     print(repr(line[:15]))
...     break
'<!doctype html>'
```

--------------------------------

### Stream to Digital Ocean Spaces

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes bytes to a Digital Ocean Spaces bucket using credentials from a specified boto3 profile and a custom endpoint URL.

```python
# Stream to Digital Ocean Spaces bucket providing credentials from boto3 profile
session = boto3.Session(profile_name="digitalocean")
client = session.client("s3", endpoint_url="https://ams3.digitaloceanspaces.com")
transport_params = {"client": client}
with open("s3://bucket/key.txt", "wb", transport_params=transport_params) as fout:
    fout.write(b"here we stand")
```

--------------------------------

### GCS Advanced Usage with Blob Open Kwargs and Properties

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Propagate keyword arguments to GCS open and set blob properties before upload using transport parameters.

```python
open_kwargs = {"predefined_acl": "authenticated-read"}
properties = {"metadata": {"version": 2}, "storage_class": "COLDLINE"}
fout = open(
    "gcs://bucket/key",
    "wb",
    transport_params={"blob_open_kwargs": open_kwargs, "blob_properties": properties},
)
```

--------------------------------

### Access S3 Object Properties with boto3

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Retrieve object properties like content type by converting a smart_open file object to a boto3 S3 Object. This method is applicable only when reading and writing via S3.

```python
import boto3
from smart_open import open
resource = boto3.resource('s3')  # Pass additional resource parameters here
with open('s3://commoncrawl/robots.txt') as fin:
   print(fin.readline().rstrip())
   boto3_s3_object = fin.to_boto3(resource)
   print(repr(boto3_s3_object))
   print(boto3_s3_object.content_type)  # Using the boto3 API here
```

--------------------------------

### Read Compressed File with Compression Disabled

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads a file explicitly disabling compression, even if the extension suggests it. Use 'rb' mode for binary data.

```python
>>> from smart_open import open
>>> with open('tests/test_data/1984.txt.gz', 'rb', compression='disable') as fin:
...     print(fin.read(32))
b'\x1f\x8b\x08\x08\x85F\x94\\x00\x031984.txt\x005\x8f=r\xc3@\x08\x85{\x9d\xe2\x1d@'

```

--------------------------------

### Access S3 Object with Custom Buffer Size

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Opens an object from AWS S3 using `smart_open`, specifying a custom buffer size for the transport layer via `transport_params`. Adjust buffer size based on performance needs.

```python
>>> fin = open('s3://commoncrawl/robots.txt', transport_params=dict(buffer_size=1024))

```

--------------------------------

### Update S3 URI Scheme for Version IDs

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

S3 URIs now recognize '?versionId=...' query parameters, forwarding them to 'smart_open.s3.open' as 'version_id'. This matches conventions used by s3fs and the AWS CLI.

```diff
- smart_open.open('s3://bucket/key', transport_params={'version_id': 'v1'})
+ smart_open.open('s3://bucket/key?versionId=v1')
```

--------------------------------

### Accessing Underlying boto3 S3 Object

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Retrieve the underlying boto3 S3 object using the `to_boto3()` method for direct interaction with the boto3 API. This is useful for performing operations not directly supported by smart_open.

```python
import boto3
from smart_open import open
resource = boto3.resource('s3')  # Pass additional resource parameters here
with open('s3://commoncrawl/robots.txt') as fin:
    boto3_object = fin.to_boto3(resource)
    print(boto3_object)
    print(boto3_object.get()['LastModified'])

```

--------------------------------

### Stream from Custom S3 Server

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads lines from an S3 bucket using a custom S3 server endpoint and provided credentials.

```python
# stream from a completely custom s3 server, like s3proxy:
client = boto3.client(
    "s3", endpoint_url="http://host:port", aws_access_key_id="user", aws_secret_access_key="secret"
)
for line in open("s3://mybucket/mykey.txt", transport_params={"client": client}):
    print(line)
```

--------------------------------

### Stream Lines from HDFS

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads and prints lines from a file stored in HDFS, specifying UTF-8 encoding.

```python
# stream from HDFS
for line in open("hdfs://host:port/user/hadoop/my_file.txt", encoding="utf8"):
    print(line)
```

--------------------------------

### Stream Lines from Azure Blob Storage

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads and prints lines from a blob in Azure Blob Storage using a connection string to authenticate.

```python
# stream from Azure Blob Storage
connect_str = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
transport_params = {
    "client": azure.storage.blob.BlobServiceClient.from_connection_string(connect_str),
}
for line in open("azure://mycontainer/myfile.txt", transport_params=transport_params):
    print(line)
```

--------------------------------

### Efficient S3 Write with Temporary File Buffer

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

This method reduces memory usage by using a temporary file as a buffer for S3 uploads. It incurs additional disk I/O, which may be slower than in-memory buffering.

```python
import boto3
from smart_open import open

with tempfile.NamedTemporaryFile() as tmp:
    transport_params = {"writebuffer": tmp}
    with open("s3://bucket/key", "w", transport_params=transport_params) as fout:
        fout.write(lots_of_data)
```

--------------------------------

### Azure Blob Storage Authentication with BlobServiceClient

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Authenticate with Azure Blob Storage by passing an azure.storage.blob.BlobServiceClient object as a transport parameter. Ensure the AZURE_STORAGE_CONNECTION_STRING environment variable is set.

```python
import os
from azure.storage.blob import BlobServiceClient

azure_storage_connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
client = BlobServiceClient.from_connection_string(azure_storage_connection_string)
fin = open("azure://my_container/my_blob.txt", transport_params={"client": client})
```

--------------------------------

### Pass boto3.Session Object Instead of profile_name

Source: https://github.com/piskvorky/smart_open/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.md

The `profile_name` parameter is removed. Pass a `boto3.Session` object within the `transport_params` dictionary instead.

```diff
smart_open.smart_open(url, 'r', profile_name='smart_open').read(32)
+ params = {'session': boto3.Session(profile_name='smart_open')}
+ open(url, 'r', transport_params=params).read(32)
```

--------------------------------

### Stream Content into HDFS

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes bytes to a file in HDFS.

```python
# stream content *into* HDFS (write mode):
with open("hdfs://host:port/user/hadoop/my_file.txt", "wb") as fout:
    fout.write(b"hello world")
```

--------------------------------

### Stream Lines from S3 Object

Source: https://github.com/piskvorky/smart_open/blob/develop/help.txt

Iterate over lines from a file stored in an S3 bucket. This snippet demonstrates basic S3 integration.

```python
>>> from smart_open import open
>>> 
>>> # stream lines from an S3 object
>>> for line in open('s3://commoncrawl/robots.txt'):
...    print(repr(line))
...    break
'User-Agent: *\n'
```

--------------------------------

### Stream Lines from WebHDFS

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads and prints lines from a file stored in WebHDFS.

```python
# stream from WebHDFS
for line in open("webhdfs://host:port/user/hadoop/my_file.txt"):
    print(line)
```

--------------------------------

### Efficient S3 Write with Smaller Part Size

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Use this method to reduce memory usage during S3 multipart uploads by specifying a smaller part size. Ensure the part size is at least 5MB and does not exceed AWS's limit of 10,000 parts per upload.

```python
import boto3
from smart_open import open

tp = {"min_part_size": 5 * 1024**2}
with open("s3://bucket/key", "w", transport_params=tp) as fout:
    fout.write(lots_of_data)
```

--------------------------------

### Single-Part Upload to S3 with Temporary File Buffer

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes bytes to an S3 object using a single-part upload and a tempfile.TemporaryFile as the write buffer to reduce memory footprint.

```python
# now with tempfile.TemporaryFile instead of the default io.BytesIO (to reduce memory footprint)
import tempfile

with (
    tempfile.TemporaryFile() as tmp,
    open(
        "s3://smart-open-py37-benchmark-results/test.txt",
        "wb",
        transport_params={"multipart_upload": False, "writebuffer": tmp},
    ) as fout,
):
    bytes_written = fout.write(b"hello world!")
    print(bytes_written)
```

--------------------------------

### Stream Content into S3 with Custom Client

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Writes bytes to an S3 object using a custom boto3 client with specific configurations for connection pooling, keep-alive, and retries.

```python
import os, boto3, botocore
from smart_open import open

# stream content *into* S3 (write mode) using a custom client
# this client is thread-safe ref https://github.com/boto/boto3/blob/1.38.41/docs/source/guide/clients.rst?plain=1#L111
config = botocore.client.Config(
    max_pool_connections=64,
    tcp_keepalive=True,
    retries={"max_attempts": 6, "mode": "adaptive"},
)
client = boto3.Session(
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
).client("s3", config=config)
with open(
    "s3://smart-open-py37-benchmark-results/test.txt", "wb", transport_params={"client": client}
) as fout:
    bytes_written = fout.write(b"hello world!")
    print(bytes_written)
```

--------------------------------

### Accessing Underlying boto3 Object for Versioned S3 Objects

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

When working with versioned S3 objects, `to_boto3()` returns a specific `s3.ObjectVersion` instance, allowing access to version-specific metadata and operations.

```python
from smart_open import open
resource = boto3.resource('s3')
params = {'version_id': 'KiQpZPsKI5Dm2oJZy_RzskTOtl2snjBg'}
with open('s3://smart-open-versioned/demo.txt', transport_params=params) as fin:
    print(fin.to_boto3(resource))

```

--------------------------------

### Stream Lines from GCS

Source: https://github.com/piskvorky/smart_open/blob/develop/README.md

Reads and prints lines from a file stored in Google Cloud Storage (GCS).

```python
# stream from GCS
for line in open("gcs://my_bucket/my_file.txt"):
    print(line)
```

--------------------------------

### Specify Request Payer for S3 Access

Source: https://github.com/piskvorky/smart_open/blob/develop/HOWTO.md

Configure smart_open to access public S3 buckets where the requester must pay for data access. This is done by passing specific transport parameters to the open function.

```python
>>> from smart_open import open
>>> params = {'client_kwargs': {'S3.Client.get_object': {'RequestPayer': 'requester'}}}
>>> with open('s3://arxiv/pdf/arXiv_pdf_manifest.xml', transport_params=params) as fin:
...    print(fin.readline())
<?xml version='1.0' standalone='yes'?>
<BLANKLINE>
```