Joblib

https://joblib.readthedocs.io/en/stable
Admin
Joblib is a set of tools for lightweight pipelining in Python, providing transparent disk-caching of...
Tokens:62,740
Snippets:730
Trust Score:8.3
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install and Set Up Pre-commit Hooks

Source: https://joblib.readthedocs.io/en/stable/developing.html

Install the pre-commit tool and set up its hooks for code style checks before each commit.

```bash
pip install pre-commit
pre-commit install
```

--------------------------------

### Install Joblib for All Users (Manual)

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

Install joblib for all users by running this command from the expanded tarball directory. Administrator privileges are required.

```bash
python -m pip install .
```

--------------------------------

### Install Documentation Dependencies

Source: https://joblib.readthedocs.io/en/stable/developing.html

Install the necessary dependencies for building the documentation using pip. Requires sphinx (>=1.4).

```bash
pip install -U -r .readthedocs-requirements.txt

```

--------------------------------

### Install Joblib with Pip (All Users)

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

Use this command to install joblib for all users on the system. Administrator privileges may be required.

```bash
pip install joblib
```

--------------------------------

### Install Joblib in Editable Mode

Source: https://joblib.readthedocs.io/en/stable/developing.html

Install Joblib in editable mode from the source directory for development.

```bash
pip install -e .
```

--------------------------------

### Setup Dask Distributed Client

Source: https://joblib.readthedocs.io/en/stable/_downloads/fec4ba264e936c334617fdc79f61159e/distributed_backend_simple.ipynb

Initializes a local Dask cluster and connects a client to it. This setup is essential for using Dask as a backend for parallel processing. The Dask dashboard link is printed for monitoring.

```python
from dask.distributed import Client, LocalCluster

# replace with whichever cluster class you're using
# https://docs.dask.org/en/stable/deploying.html#distributed-computing
cluster = LocalCluster()
# connect client to your cluster
client = Client(cluster)

# Monitor your computation with the Dask dashboard
print(client.dashboard_link)
```

--------------------------------

### Install Joblib for All Users (Unix Prefix)

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

On Unix systems, install joblib for all users into the /usr/local directory to prevent conflicts with system packages. Administrator privileges are required.

```bash
python -m pip install --prefix /usr/local .
```

--------------------------------

### Install Joblib with Pip (System Prefix)

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

On Unix-like systems, install joblib to a specific prefix like /usr/local to avoid interfering with system packages. Administrator privileges may be required.

```bash
pip install --prefix /usr/local joblib
```

--------------------------------

### Install Joblib from Local Environment

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

Install joblib within a local environment after expanding the tarball. This method is recommended for isolating changes to your account and avoids requiring administrator rights.

```bash
python -m pip install --user .
```

--------------------------------

### Install Joblib with Pip (User Specific)

Source: https://joblib.readthedocs.io/en/stable/_sources/installing.rst.txt

Install joblib only for the current user. This is a convenient option that does not require administrator privileges.

```bash
pip install --user joblib
```

--------------------------------

### Basic Object Persistence with Joblib

Source: https://joblib.readthedocs.io/en/stable/persistence.html

Shows a simple example of creating an object, saving it to a file using joblib.dump, and then reloading it using joblib.load.

```python
from tempfile import mkdtemp
savedir = mkdtemp()
import os
filename = os.path.join(savedir, 'test.joblib')

```

```python
import numpy as np
to_persist = [('a', [1, 2, 3]), ('b', np.arange(10))]

```

```python
import joblib
joblib.dump(to_persist, filename)

```

```python
joblib.load(filename)

```

--------------------------------

### Joblib Parallel pre_dispatch Example

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html

Illustrates using 'pre_dispatch' in a producer-consumer scenario where data is generated on the fly. The producer is called multiple times before the parallel loop starts and then to generate new data.

```python
from math import sqrt
from joblib import Parallel, delayed
def producer():
    for i in range(6):
        print('Produced %s' % i)
        yield i
out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')( 
    delayed(sqrt)(i) for i in producer())
Produced 0
Produced 1
Produced 2
[Parallel(n_jobs=2)]: Done 1 jobs     | elapsed:  0.0s
Produced 3
[Parallel(n_jobs=2)]: Done 2 jobs     | elapsed:  0.0s
Produced 4
[Parallel(n_jobs=2)]: Done 3 jobs     | elapsed:  0.0s
Produced 5
[Parallel(n_jobs=2)]: Done 4 jobs     | elapsed:  0.0s
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed:  0.0s remaining: 0.0s
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed:  0.0s finished
```

--------------------------------

### Registering a Custom Backend with Joblib

Source: https://joblib.readthedocs.io/en/stable/_sources/custom_parallel_backend.rst.txt

This example shows how to register a custom backend named 'custom' by importing a library and then using it within `joblib.parallel_config`. It highlights the failure before registration and success after.

```python
>>> import joblib
>>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
...     ...  # this fails
KeyError: 'custom'

# Import library to register external backend
>>> import my_custom_backend_library  # doctest: +SKIP
>>> with joblib.parallel_config(backend='custom'):  # doctest: +SKIP
...     ... # this works
```

--------------------------------

### Fetch and load dataset

Source: https://joblib.readthedocs.io/en/stable/_downloads/eb580176dbb28422a2948502cec12406/compressors_comparison.ipynb

Fetches a dataset from a URL and loads it into a pandas DataFrame. Ensure pandas is installed.

```python
import pandas as pd

url = "https://github.com/joblib/dataset/raw/main/kddcup.data.gz"
names = (
    "duration, protocol_type, service, flag, src_bytes, "
    "dst_bytes, land, wrong_fragment, urgent, hot, "
    "num_failed_logins, logged_in, num_compromised, "
    "root_shell, su_attempted, num_root, "
    "num_file_creations, "
).split(", ")

data = pd.read_csv(url, names=names, nrows=1e6)
```

--------------------------------

### Setup Joblib Memory for Caching

Source: https://joblib.readthedocs.io/en/stable/_downloads/58796f572e8ca7b503da779777847d75/nested_parallel_memory.ipynb

Initializes joblib.Memory to cache the results of a function. Specify the cache directory and set verbose level.

```python
from joblib import Memory

location = "./cachedir"
memory = Memory(location, verbose=0)
costly_compute_cached = memory.cache(costly_compute)
```

--------------------------------

### Registering a custom compressor

Source: https://joblib.readthedocs.io/en/stable/_sources/generated/joblib.register_compressor.rst.txt

Example of how to define and register a custom compressor using `register_compressor`. This involves creating a class with `compress` and `decompress` methods and then registering it.

```python
from joblib.compressor import register_compressor

class MyCompressor:
    def compress(self, data):
        # Implement custom compression logic
        return compressed_data

    def decompress(self, compressed_data):
        # Implement custom decompression logic
        return data

register_compressor('my_compressor', MyCompressor)

# Now you can use 'my_compressor' when saving joblib objects
# from joblib import dump, load
# dump(my_object, 'my_file.joblib', compress='my_compressor')
```

--------------------------------

### Using Ray backend for parallel processing

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.parallel_backend.html

This example shows how to configure and use the 'ray' joblib backend for distributing workloads to a cluster of nodes. The 'register_ray()' function must be called first to enable the backend.

```python
from ray.util.joblib import register_ray

register_ray()
with parallel_backend("ray"):
    print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
```

--------------------------------

### Using parallel_backend with Dask

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.parallel_backend.html

This example illustrates how to use the `parallel_backend` context manager with the 'dask' backend for distributed computing. It includes setting up a Dask client and passing data to be scattered across the cluster.

```APIDOC
## Using joblib.parallel_backend with Dask

### Description
This snippet demonstrates using `parallel_backend` with the 'dask' backend for distributed computation across multiple machines. It requires a Dask cluster to be set up.

### Method
Context Manager (`with` statement)

### Endpoint
N/A (Python context manager)

### Parameters
#### Context Manager Parameters
- **backend** (string) - Required - Set to 'dask'.
- **scatter** (list) - Optional - Data to be scattered to the Dask workers.
- **n_jobs** (int) - Optional - The number of parallel jobs to run. Defaults to -1.
- **inner_max_num_threads** (int) - Optional - Limit for thread pools in third-party libraries within workers.
- **backend_params** (**kwargs) - Optional - Additional parameters specific to the Dask backend.

### Request Example
```python
import joblib
from sklearn.model_selection import GridSearchCV
from dask.distributed import Client, LocalCluster

# Assume estimator, param_grid, X, y are defined
# create a local Dask cluster
cluster = LocalCluster()
client = Client(cluster)

grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)

with joblib.parallel_backend("dask", scatter=[X, y]):
    grid_search.fit(X, y)
```

### Response
#### Success Response
Results of the computation performed using Dask workers.

#### Response Example
(Depends on the `grid_search.fit` operation)
```

--------------------------------

### Using parallel_backend with Ray

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.parallel_backend.html

This example shows how to integrate joblib with the 'ray' backend for distributed workloads. It involves registering the Ray backend and then using `parallel_backend` within a `with` block.

```APIDOC
## Using joblib.parallel_backend with Ray

### Description
This snippet demonstrates how to use joblib with the 'ray' backend for distributing workloads across a cluster of nodes managed by Ray.

### Method
Context Manager (`with` statement)

### Endpoint
N/A (Python context manager)

### Parameters
#### Context Manager Parameters
- **backend** (string) - Required - Set to 'ray'.
- **n_jobs** (int) - Optional - The number of parallel jobs to run. Defaults to -1.
- **inner_max_num_threads** (int) - Optional - Limit for thread pools in third-party libraries within workers.
- **backend_params** (**kwargs) - Optional - Additional parameters specific to the Ray backend.

### Request Example
```python
from ray.util.joblib import register_ray
from joblib import Parallel, delayed, parallel_backend
from operator import neg

register_ray()

with parallel_backend("ray"):
    print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
```

### Response
#### Success Response
Results of the parallel computation executed on the Ray cluster.

#### Response Example
```
[-1, -2, -3, -4, -5]
```
```

--------------------------------

### Using parallel_backend with a string backend name

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.parallel_backend.html

This example demonstrates how to use the `parallel_backend` context manager by specifying the backend name as a string. It shows the usage with the 'threading' backend and executing a simple parallel operation.

```APIDOC
## Using joblib.parallel_backend with a string backend name

### Description
This snippet shows how to use the `parallel_backend` context manager to switch to the 'threading' backend for parallel execution.

### Method
Context Manager (`with` statement)

### Endpoint
N/A (Python context manager)

### Parameters
#### Context Manager Parameters
- **backend** (string) - Required - The name of the backend to use (e.g., 'threading', 'loky', 'dask', 'ray').
- **n_jobs** (int) - Optional - The number of parallel jobs to run. Defaults to -1 (all available workers).
- **inner_max_num_threads** (int) - Optional - Limit for thread pools in third-party libraries within workers.
- **backend_params** (**kwargs) - Optional - Additional parameters specific to the chosen backend.

### Request Example
```python
from operator import neg
from joblib import Parallel, delayed, parallel_backend

with parallel_backend('threading'):
    print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
```

### Response
#### Success Response
Output of the parallel execution.

#### Response Example
```
[-1, -2, -3, -4, -5]
```
```

--------------------------------

### Register and Use Ray Backend in Joblib

Source: https://joblib.readthedocs.io/en/stable/parallel.html

This snippet demonstrates how to register the Ray backend with Joblib and then use it within a `parallel_config` context manager to execute parallel tasks. Ensure Ray is installed and configured.

```python
>>> from ray.util.joblib import register_ray
>>> register_ray()
>>> with parallel_config(backend="ray"):
...     print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
[-1, -2, -3, -4, -5]
```

--------------------------------

### Joblib Parallel Traceback Example

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html

Demonstrates how joblib.Parallel displays tracebacks from child processes, including the line of error and parameter values.

```python
from heapq import nlargest
from joblib import Parallel, delayed
Parallel(n_jobs=2)(
    delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
-----------------------------------------------------------------------
Sub-process traceback:
-----------------------------------------------------------------------
TypeError                                      Mon Nov 12 11:37:46 2012
PID: 12934                                Python 2.7.3: /usr/bin/python
........................................................................
/usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
    419         if n >= size:
    420             return sorted(iterable, key=key, reverse=True)[:n]
    421
    422     # When key is none, use simpler decoration
    423     if key is None:
--> 424         it = izip(iterable, count(0,-1))           # decorate
    425         result = _nlargest(n, it) 
    426         return map(itemgetter(0), result)          # undecorate
    427
    428     # General case, slowest method
 TypeError: izip argument #1 must support iteration
_______________________________________________________________________
```

--------------------------------

### Dask Backend Execution Output

Source: https://joblib.readthedocs.io/en/stable/auto_examples/parallel/distributed_backend_simple.html

Example output showing the DaskDistributedBackend being used by joblib.Parallel, including task completion times and batch size adjustments.

```text
[Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 2 concurrent workers.
[Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    0.1s
[Parallel(n_jobs=-1)]: Batch computation too fast (0.1356656551361084s.) Setting batch_size=2.
[Parallel(n_jobs=-1)]: Done   2 tasks      | elapsed:    0.1s
[Parallel(n_jobs=-1)]: Done   3 tasks      | elapsed:    0.2s
[Parallel(n_jobs=-1)]: Done   4 tasks      | elapsed:    0.2s
[Parallel(n_jobs=-1)]: Done   6 tasks      | elapsed:    0.4s
[Parallel(n_jobs=-1)]: Done   8 out of  10 | elapsed:    0.4s remaining:    0.1s
[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.5s finished

```

--------------------------------

### Run Joblib Test Suite

Source: https://joblib.readthedocs.io/en/stable/developing.html

Execute the test suite for Joblib using pytest. Ensure pytest (version >= 3) and coverage modules are installed.

```bash
pytest joblib

```

--------------------------------

### Parallel Computation with Joblib

Source: https://joblib.readthedocs.io/en/stable/index.html

Utilize Parallel and delayed to execute a list of operations in parallel. This example demonstrates computing the square root of squared numbers efficiently.

```python
>>> from joblib import Parallel, delayed
>>> from math import sqrt
>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
```

--------------------------------

### Basic Object Persistence with Joblib

Source: https://joblib.readthedocs.io/en/stable/_sources/persistence.rst.txt

Saves a Python list containing a string and a NumPy array to a file using joblib.dump and reloads it using joblib.load. Requires temporary directory setup.

```python
from tempfile import mkdtemp
savedir = mkdtemp()
import os
filename = os.path.join(savedir, 'test.joblib')
import numpy as np
to_persist = [('a', [1, 2, 3]), ('b', np.arange(10))]
import joblib
joblib.dump(to_persist, filename)  # doctest: +ELLIPSIS
# ['...test.joblib']
joblib.load(filename)
# [('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Parallel execution with multiprocessing backend and pickle

Source: https://joblib.readthedocs.io/en/stable/_downloads/6f54ccb28e994a64c34b08c8f124a046/serialization_and_wrappers.ipynb

Compares serialization speed by using the 'multiprocessing' backend with standard pickle serialization. This is only executed if the start method is not 'spawn' and is generally faster for large objects but has platform limitations.

```python
import multiprocessing as mp

if mp.get_start_method() != "spawn":

    def func_async(i, *args):
        return 2 * i

    with parallel_config("multiprocessing"):
        t_start = time.time()
        Parallel(n_jobs=2)(delayed(func_async)(21, large_list) for _ in range(1))
        print(
            "With multiprocessing backend and pickle serialization: {:.3f}s".format(
                time.time() - t_start
            )
        )
```

--------------------------------

### Build Joblib Documentation

Source: https://joblib.readthedocs.io/en/stable/developing.html

Build the HTML documentation for Joblib using the make command. The output will be in the doc/_build/html directory.

```bash
make doc

```

--------------------------------

### Get CPU Count

Source: https://joblib.readthedocs.io/en/stable/_sources/generated/joblib.cpu_count.rst.txt

This snippet shows how to import and use the `cpu_count` function from the joblib library to get the number of available CPU cores.

```python
from joblib import cpu_count

print(f"Number of CPU cores: {cpu_count()}")
```

--------------------------------

### MemoryMonitor Helper Class

Source: https://joblib.readthedocs.io/en/stable/_downloads/db658a25107c199addf4f0b83d7dc5ac/parallel_generator.ipynb

Monitors memory usage in MB in a separate thread. It tracks the memory of a process and its children. Ensure psutil is installed (`pip install psutil`).

```python
import time
from threading import Thread

from psutil import Process


class MemoryMonitor(Thread):
    """Monitor the memory usage in MB in a separate thread.

    Note that this class is good enough to highlight the memory profile of
    Parallel in this example, but is not a general purpose profiler fit for
    all cases.
    """

    def __init__(self):
        super().__init__()
        self.stop = False
        self.memory_buffer = []
        self.start()

    def get_memory(self):
        "Get memory of a process and its children."
        p = Process()
        memory = p.memory_info().rss
        for c in p.children():
            memory += c.memory_info().rss
        return memory

    def run(self):
        memory_start = self.get_memory()
        while not self.stop:
            self.memory_buffer.append(self.get_memory() - memory_start)
            time.sleep(0.2)

    def join(self):
        self.stop = True
        super().join()
```

--------------------------------

### Get a reference to a cached result

Source: https://joblib.readthedocs.io/en/stable/memory.html

Use `call_and_shelve` to store the result of a function call on disk and get a reference to it. This is useful for dispatching large data across workers without sending the data itself.

```python
>>> result = g.call_and_shelve(4)
A long-running calculation, with parameter 4
>>> result
MemorizedResult(location="...", func="...g...", args_id="...")
```

--------------------------------

### MemoryMonitor Helper Class

Source: https://joblib.readthedocs.io/en/stable/_sources/auto_examples/parallel_generator.rst.txt

Monitors memory usage in a separate thread. Requires psutil to be installed.

```Python
import time
from threading import Thread

from psutil import Process


class MemoryMonitor(Thread):
    """Monitor the memory usage in MB in a separate thread.

    Note that this class is good enough to highlight the memory profile of
    Parallel in this example, but is not a general purpose profiler fit for
    all cases.
    """

    def __init__(self):
        super().__init__()
        self.stop = False
        self.memory_buffer = []
        self.start()

    def get_memory(self):
        """Get memory of a process and its children."""
        p = Process()
        memory = p.memory_info().rss
        for c in p.children():
            memory += c.memory_info().rss
        return memory

    def run(self):
        memory_start = self.get_memory()
        while not self.stop:
            self.memory_buffer.append(self.get_memory() - memory_start)
            time.sleep(0.2)

    def join(self):
        self.stop = True
        super().join()
```

--------------------------------

### Clone Joblib Repository

Source: https://joblib.readthedocs.io/en/stable/developing.html

Use this command to get the latest source code of Joblib using Git.

```bash
git clone https://github.com/joblib/joblib.git
```

--------------------------------

### Make Release and Upload to PyPI

Source: https://joblib.readthedocs.io/en/stable/developing.html

Create a release package (sdist and wheel) and upload it to PyPI. This command is intended for project managers.

```bash
pip install build
python -m build --sdist --wheel
twine upload dist/*

```

--------------------------------

### Dump with LZ4 Compression

Source: https://joblib.readthedocs.io/en/stable/persistence.html

If the `lz4` package is installed, it can be used automatically for compression by `joblib.dump()`. Requires Python 3.

```python
>>> joblib.dump(to_persist, filename + '.lz4')
['...test.joblib.lz4']
>>> joblib.load(filename + '.lz4')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Retrieve a cached value

Source: https://joblib.readthedocs.io/en/stable/memory.html

Access the actual cached data using the `get` method on the `MemorizedResult` object. This reads the value from disk.

```python
>>> result.get()
array([0.08, 0.77, 0.77, 0.08])
```

--------------------------------

### Using a registered custom backend

Source: https://joblib.readthedocs.io/en/stable/custom_parallel_backend.html

Shows a successful execution after importing a library that registers the 'custom' backend, making it available for use with `joblib.parallel_config`.

```python
# Import library to register external backend
>>> import my_custom_backend_library
>>> with joblib.parallel_config(backend='custom'):
...     ... # this works
```

--------------------------------

### Create Large NumPy Array

Source: https://joblib.readthedocs.io/en/stable/_downloads/a0562d33325f994d06f0e84c36489eb6/parallel_memmap.ipynb

Generates a large random NumPy array and defines slices for processing. This is the initial data setup.

```python
import numpy as np

data = np.random.random((int(1e7),))
window_size = int(5e5)



slices = [
    slice(start, start + window_size)
    for start in range(0, data.size - window_size, int(1e5))
]
```

--------------------------------

### Persistence with gzip.GzipFile Object

Source: https://joblib.readthedocs.io/en/stable/_sources/persistence.rst.txt

Demonstrates using joblib.dump and joblib.load with a gzip.GzipFile object, specifying a compression level. This shows integration with standard library compression modules.

```python
import gzip
with gzip.GzipFile(filename + '.gz', 'wb', compresslevel=3) as fo:  # doctest: +ELLIPSIS
   joblib.dump(to_persist, fo)
with gzip.GzipFile(filename + '.gz', 'rb') as fo:  # doctest: +ELLIPSIS
   joblib.load(fo)
# [('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Resetting Loky Pickler

Source: https://joblib.readthedocs.io/en/stable/_downloads/6f54ccb28e994a64c34b08c8f124a046/serialization_and_wrappers.ipynb

Resets the loky pickler to avoid interference with subsequent examples in sphinx-gallery. This is a utility function for testing environments.

```python
# Reset the loky_pickler to avoid border effects with other examples in
# sphinx-gallery.
set_loky_pickler()
```

--------------------------------

### Clear a cached value

Source: https://joblib.readthedocs.io/en/stable/memory.html

Use the `clear` method to remove the cached value from disk. Subsequent calls to `get` for this reference will raise a `KeyError`.

```python
>>> result.clear()
>>> result.get()
Traceback (most recent call last):
... 
KeyError: 'Non-existing cache value (may have been cleared).\nFile ... does not exist'
```

--------------------------------

### Dump with Compression Method Name (Default Level)

Source: https://joblib.readthedocs.io/en/stable/persistence.html

Provide the compression method name as a string to `compress`. This uses the default compression level for that method.

```python
>>> joblib.dump(to_persist, filename + '.gz', compress='gzip')
['...test.joblib.gz']
```

--------------------------------

### Decorating Methods at Instantiation Time

Source: https://joblib.readthedocs.io/en/stable/_sources/memory.rst.txt

Demonstrates the correct way to cache a method by decorating it at instantiation time, as decorating at class definition is not supported.

```python
class Foo(object):

    def __init__(self, args):
        self.method = memory.cache(self.method)

    def method(self, ...):
        pass
```

--------------------------------

### Parallel Execution with Progress Meter

Source: https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html

Illustrates how to enable and observe a progress meter during parallel execution by setting the 'verbose' parameter. Higher 'verbose' values provide more detailed progress updates.

```python
from time import sleep
from joblib import Parallel, delayed
r = Parallel(n_jobs=2, verbose=10)(
    delayed(sleep)(.2) for _ in range(10))
[Parallel(n_jobs=2)]: Done   1 tasks      | elapsed:    0.6s
[Parallel(n_jobs=2)]: Done   4 tasks      | elapsed:    0.8s
[Parallel(n_jobs=2)]: Done  10 out of  10 | elapsed:    1.4s finished
```

--------------------------------

### Load LZ4 Compressed Data

Source: https://joblib.readthedocs.io/en/stable/_downloads/eb580176dbb28422a2948502cec12406/compressors_comparison.ipynb

Measures the time taken to load data that was previously compressed using LZ4. Requires the 'lz4' package to be installed.

```python
start = time.time()
with open(pickle_file, "rb") as f:
    load(f)
lz4_load_duration = time.time() - start
print("LZ4 load duration: %0.3fs" % lz4_load_duration)
```

--------------------------------

### Attempting to use an unregistered custom backend

Source: https://joblib.readthedocs.io/en/stable/custom_parallel_backend.html

Demonstrates the `KeyError` that occurs when trying to use a custom backend that has not been registered with joblib.

```python
>>> import joblib
>>> with joblib.parallel_config(backend='custom'):
...     ...  # this fails
KeyError: 'custom'
```

--------------------------------

### Dump Data with LZ4 Compression

Source: https://joblib.readthedocs.io/en/stable/_downloads/eb580176dbb28422a2948502cec12406/compressors_comparison.ipynb

Measures the time taken to dump data using the LZ4 compression method. Requires the 'lz4' package to be installed.

```python
start = time.time()
with open(pickle_file, "wb") as f:
    dump(data, f, compress="lz4")
lz4_dump_duration = time.time() - start
print("LZ4 dump duration: %0.3fs" % lz4_dump_duration)
```

--------------------------------

### Setting Up Nested Parallelism Backend

Source: https://joblib.readthedocs.io/en/stable/_sources/custom_parallel_backend.rst.txt

Implement `get_nested_backend` to define the default backend for nested `Parallel` calls. This helps manage resource allocation and avoid excessive thread spawning.

```python
def get_nested_backend(self):
    """Backend instance to be used by nested Parallel calls.

    By default a thread-based backend is used for the first level of
    nesting. Beyond, switch to sequential backend to avoid spawning too
    many threads on the host.
    """
    nesting_level = getattr(self, "nesting_level", 0) + 1
    return LokyBackend(nesting_level=nesting_level), None
```

--------------------------------

### Parallel execution with 'loky' backend

Source: https://joblib.readthedocs.io/en/stable/_downloads/715d8eefe44aa2b912e10d4c21fb08de/parallel_random_state.ipynb

Demonstrates parallel execution using the 'loky' backend, which is the default and generally safe for random state management.

```python
backend = "loky"
random_vector = Parallel(n_jobs=2, backend=backend)(
    delayed(stochastic_function)(10) for _ in range(n_vectors)
)
print_vector(random_vector, backend)
```

--------------------------------

### Configuring joblib Backend with parallel_config

Source: https://joblib.readthedocs.io/en/stable/_sources/parallel.rst.txt

Demonstrates using the joblib.parallel_config context manager to set the backend to 'threading' and specify the number of jobs. This allows for flexible backend selection and parameter tuning.

```python
from joblib import parallel_config
with parallel_config(backend='threading', n_jobs=2):
   Parallel()(delayed(sqrt)(i ** 2) for i in range(10))
```

--------------------------------

### Compressed Persistence with LZ4

Source: https://joblib.readthedocs.io/en/stable/_sources/persistence.rst.txt

Saves an object to an LZ4 compressed file. LZ4 compression is automatically available if the 'lz4' package is installed and the Python version is >= 3.

```python
joblib.dump(to_persist, filename + '.lz4')  # doctest: +ELLIPSIS
# ['...test.joblib.lz4']
joblib.load(filename + '.lz4')
# [('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Parallel Processing with return_as='list'

Source: https://joblib.readthedocs.io/en/stable/_downloads/db658a25107c199addf4f0b83d7dc5ac/parallel_generator.ipynb

Demonstrates parallel task execution where results are collected into a list. Memory usage can grow linearly with the number of tasks.

```python
from joblib import Parallel, delayed

monitor = MemoryMonitor()
print("Running tasks with return_as='list'...")
res = Parallel(n_jobs=2, return_as="list")(
    delayed(return_big_object)(i) for i in range(150)
)
print("Accumulate results:", end="")
res = accumulator_sum(res)
print("All tasks completed and reduced successfully.")

# Report memory usage
del res  # we clean the result to avoid memory border effects
monitor.join()
peak = max(monitor.memory_buffer) / 1e9
print(f"Peak memory usage: {peak:.2f}GB")
```

--------------------------------

### Parallel with return_as='generator'

Source: https://joblib.readthedocs.io/en/stable/_sources/auto_examples/parallel_generator.rst.txt

Use `return_as='generator'` to get a generator of results as they are ready. This reduces memory footprint by processing results incrementally. Requires importing `Parallel` and `delayed` from `joblib`.

```Python
from joblib import Parallel, delayed
from joblib.externals.loky import get_reдукer
from joblib.memory import MemoryMonitor
import numpy as np

def return_big_object(i):
    return np.zeros(1024 * 1024, dtype=np.int8)

def accumulator_sum(res):
    total = 0
    for r in res:
        total += r.sum()
    return total

monitor_gen = MemoryMonitor()
print("Create result generator with return_as='generator'...")
res = Parallel(n_jobs=2, return_as="generator")(delayed(return_big_object)(i) for i in range(150))
print("Accumulate results:", end="")
res = accumulator_sum(res)
print("All tasks completed and reduced successfully.")

# Report memory usage
del res  # we clean the result to avoid memory border effects
monitor_gen.join()
peak = max(monitor_gen.memory_buffer) / 1e6
print(f"Peak memory usage: {peak:.2f}MB")
```

--------------------------------

### Instantiate Memory Context

Source: https://joblib.readthedocs.io/en/stable/_sources/memory.rst.txt

Define the cache directory and instantiate a Memory context. Set verbose to 0 to suppress output during caching operations.

```python
location = 'your_cache_location_directory'
from joblib import Memory
memory = Memory(location, verbose=0)
```

--------------------------------

### Control C-level Threadpools with inner_max_num_threads

Source: https://joblib.readthedocs.io/en/stable/custom_parallel_backend.html

Support `inner_max_num_threads` for C-level threadpools like OpenMP by setting `supports_inner_max_num_threads` to `True` and accepting the argument in the constructor. Use `_prepare_worker_env` for setup.

```python
# Backend should set the supports_inner_max_num_threads class attribute to True
# and accept the argument in the constructor to set this up in the workers.
# A helper to set this in the workers is to use environment variables provided by self._prepare_worker_env(n_jobs).
```

--------------------------------

### Persistence with File Objects

Source: https://joblib.readthedocs.io/en/stable/_sources/persistence.rst.txt

Demonstrates using joblib.dump and joblib.load with file objects (opened in binary mode) instead of filenames. This is useful for streaming data.

```python
with open(filename, 'wb') as fo:  # doctest: +ELLIPSIS
   joblib.dump(to_persist, fo)
with open(filename, 'rb') as fo:  # doctest: +ELLIPSIS
   joblib.load(fo)
# [('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Custom Backend with Constructor Arguments

Source: https://joblib.readthedocs.io/en/stable/_sources/custom_parallel_backend.rst.txt

Define a custom backend class that accepts mandatory constructor parameters like endpoint and API key. Register this backend for use with Joblib's Parallel.

```python
class MyCustomBackend(ParallelBackendBase):

    def __init__(self, endpoint, api_key, nesting_level=0):
       super().__init__(nesting_level=nesting_level)
       self.endpoint = endpoint
       self.api_key = api_key

    ... # Do something with self.endpoint and self.api_key somewhere in
    # one of the method of the class

register_parallel_backend('custom', MyCustomBackend)
```

```python
with parallel_config(backend='custom', endpoint='http://compute',
                         api_key='42'):
    Parallel()(delayed(some_function)(i) for i in range(10))
```

--------------------------------

### Create Large Data Array and Slices

Source: https://joblib.readthedocs.io/en/stable/_sources/auto_examples/parallel_memmap.rst.txt

Generates a large random NumPy array and defines a list of slices for processing. This setup is used to simulate a data-intensive task.

```Python
import numpy as np

data = np.random.random((int(1e7),))
window_size = int(5e5)
slices = [
    slice(start, start + window_size)
    for start in range(0, data.size - window_size, int(1e5))
]
```

--------------------------------

### Dump with Specific Compression Method and Level

Source: https://joblib.readthedocs.io/en/stable/persistence.html

Specify the compression method (e.g., 'gzip') and a compression level (e.g., 3) using a tuple for the `compress` argument.

```python
>>> # Dumping in a gzip compressed file using a compress level of 3.
>>> joblib.dump(to_persist, filename + '.gz', compress=('gzip', 3))
['...test.joblib.gz']
>>> joblib.load(filename + '.gz')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
>>> joblib.dump(to_persist, filename + '.bz2', compress=('bz2', 3))
['...test.joblib.bz2']
>>> joblib.load(filename + '.bz2')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
```

--------------------------------

### Create Source Tarball

Source: https://joblib.readthedocs.io/en/stable/developing.html

Generate a source tarball for packaging or distribution. This command creates the tarball in the dist directory and requires no extra Python standard library dependencies for installation.

```bash
pip install build
python -m build --sdist

```

--------------------------------

### Initialize joblib dump/load variables

Source: https://joblib.readthedocs.io/en/stable/_downloads/eb580176dbb28422a2948502cec12406/compressors_comparison.ipynb

Imports joblib dump and load functions and sets the filename for the pickled data.

```python
from joblib import dump, load

pickle_file = "./pickle_data.joblib"
```

--------------------------------

### Parallel Configuration Parameters

Source: https://joblib.readthedocs.io/en/stable/parallel.html

This section details the various parameters that can be used to configure the Parallel execution environment, including job management, verbosity, temporary storage, memory mapping, and backend selection.

```APIDOC
## Parallel Configuration Parameters

This section details the various parameters that can be used to configure the Parallel execution environment, including job management, verbosity, temporary storage, memory mapping, and backend selection.

### n_jobs
- **Type**: int, default=None
- **Description**: The maximum number of concurrently running jobs. If -1, all CPUs are used. If less than -1, (n_cpus + 1 + n_jobs) are used. If 1, no parallel computing is used. None is interpreted as n_jobs=1 unless under a `parallel_config()` context manager. n_jobs=0 raises a ValueError.

### verbose
- **Type**: int, default=0
- **Description**: The verbosity level for progress messages. Higher values increase message frequency. Above 50, output goes to stdout. Above 10, all iterations are reported.

### temp_folder
- **Type**: str or None, default=None
- **Description**: Folder for the pool to use for memmapping large arrays for sharing memory with workers. Defaults to `JOBLIB_TEMP_FOLDER` env var, then `/dev/shm`, then system temporary folder.

### max_nbytes
- **Type**: int, str, or None, optional, default=’1M’
- **Description**: Threshold for array size (in bytes or human-readable string like '1M') that triggers automated memory mapping. Use None to disable.

### mmap_mode
- **Type**: {None, ‘r+’, ‘r’, ‘w+’, ‘c’}, default=’r’
- **Description**: Memmapping mode for numpy arrays passed to workers. See numpy.memmap documentation for modes.

### prefer
- **Type**: str in {‘processes’, ‘threads’} or None, default=None
- **Description**: Hint to choose the default backend ('loky' for processes, 'threading' for threads). Ignored if `backend` is specified.

### require
- **Type**: ‘sharedmem’ or None, default=None
- **Description**: Hard constraint to select the backend. If 'sharedmem', the backend must be single-host and thread-based.

### inner_max_num_threads
- **Type**: int, default=None
- **Description**: Overwrites the limit on threads usable in third-party library threadpools (OpenBLAS, MKL, OpenMP). Only used with the `loky` backend.

### backend_params
- **Type**: dict
- **Description**: Additional parameters to pass to the backend constructor when the backend is a string.

### Notes
Joblib tries to limit oversubscription by limiting threads in third-party libraries. The default limit per worker is `max(cpu_count() // effective_n_jobs, 1)`, but can be overwritten by `inner_max_num_threads`.
Added in version 1.3.

### Examples
```python
>>> from operator import neg
>>> with parallel_config(backend='threading'):
...     print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
... 
[-1, -2, -3, -4, -5]
```
```

--------------------------------

### Configure inner max threads for parallel execution

Source: https://joblib.readthedocs.io/en/stable/parallel.html

This example shows how to programmatically limit the number of threads used by worker processes in Joblib's Parallel execution using `parallel_config`.

```python
from joblib import Parallel, delayed, parallel_config

with parallel_config(backend="loky", inner_max_num_threads=2):
    results = Parallel(n_jobs=4)(delayed(func)(x, y) for x, y in data)

```

--------------------------------

### Import necessary libraries

Source: https://joblib.readthedocs.io/en/stable/_downloads/eb580176dbb28422a2948502cec12406/compressors_comparison.ipynb

Imports the os and time modules for file operations and timing.

```python
import os
import os.path
import time
```

--------------------------------

### Directly Registering a Custom Backend Function

Source: https://joblib.readthedocs.io/en/stable/_sources/custom_parallel_backend.rst.txt

Demonstrates how to directly register a custom backend within Joblib's `EXTERNAL_BACKENDS` dictionary. This involves creating a registration function that handles potential import errors and assigning it to a backend name.

```python
def _register_custom():
    try:
        import my_custom_library
    except ImportError:
        raise ImportError("an informative error message")

EXTERNAL_BACKENDS['custom'] = _register_custom
```

--------------------------------

### Register a Custom Parallel Backend

Source: https://joblib.readthedocs.io/en/stable/_sources/custom_parallel_backend.rst.txt

Register a custom backend named 'custom' with a factory 'MyCustomBackend' for use with `parallel_config`.

```python
# Register the backend so it can be used with parallel_config
register_parallel_backend('custom', MyCustomBackend)
```
Tokens:
Raw