### mpire WorkerPool with 'spawn' start method (working example)

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/start_method.rst

Shows a correct implementation using the 'spawn' start method where the 'os' module is imported within the target function. This ensures that the function has access to necessary modules when executed in a new process.

```python
def working_job(folder, filename):
    import os
    return os.path.join(folder, filename)

# This will work
with WorkerPool(n_jobs=2, start_method='spawn') as pool:
    pool.map(working_job, [('folder', '0.p3'), ('folder', '1.p3')])
```

--------------------------------

### mpire WorkerPool with 'spawn' start method (failing example)

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/start_method.rst

Demonstrates a scenario where a function using the 'spawn' start method fails because the 'os' module is not imported within the function itself. This highlights the requirement to import modules within the target function when using 'spawn' or 'forkserver'.

```python
import os

def failing_job(folder, filename):
    return os.path.join(folder, filename)

# This will fail because 'os' is not copied to the child processes
with WorkerPool(n_jobs=2, start_method='spawn') as pool:
    pool.map(failing_job, [('folder', '0.p3'), ('folder', '1.p3')])
```

--------------------------------

### Install mpire using pip

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

This command installs the mpire package from PyPI using pip, the Python package installer. Ensure you have pip installed and updated for the best experience.

```bash
pip install mpire
```

--------------------------------

### Setup.py for Unittests with Spawn/Forkserver

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

Ensures that the setup call in setup.py is wrapped in an 'if __name__ == "__main__":' clause. This is a common solution to prevent unittests from restarting unexpectedly when using 'spawn' or 'forkserver' start methods.

```python
from setuptools import setup

if __name__ == '__main__':

    # Call setup and install any dependencies you have inside the if-clause
    setup(...)
```

--------------------------------

### Install mpire with Docs Dependencies

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Installs the mpire library along with its documentation dependencies. This is required to build the documentation locally.

```bash
pip install mpire[docs]
```

```bash
pip install .[docs]
```

--------------------------------

### Install Rich Progress Bar Support

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/install.rst

Installs the 'rich' library, enabling the use of rich progress bars within MPIRE for enhanced visualization of task progress.

```bash
pip install rich
```

--------------------------------

### Install MPIRE with Dashboard Support

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/install.rst

Installs the necessary dependencies, including Flask, to enable the MPIRE dashboard. This feature allows monitoring of MPIRE progress via a web browser and also changes MPIRE's license to BSD.

```bash
pip install mpire[dashboard]
```

--------------------------------

### Starting WorkerPool with Context Manager

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/setup.rst

Demonstrates the recommended way to start an mpire WorkerPool using a 'with' statement. This ensures proper cleanup of worker processes after the block finishes.

```python
from mpire import WorkerPool

# Start a pool of 4 workers
with WorkerPool(n_jobs=4) as pool:
    # Do some processing here
    pass

```

--------------------------------

### Install MPIRE via Pip

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/install.rst

Installs the MPIRE library using the pip package manager. This is the standard method for installing Python packages.

```bash
pip install mpire
```

--------------------------------

### mpire Start Methods and Dill Support

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/changelog.rst

Details the addition of support for 'spawn' and 'forkserver' start methods and optional 'dill' support via the 'multiprocess' library.

```python
Added support for using different start methods ('spawn' and 'forkserver') instead of only the default method 'fork'
Added optional support for using dill_ in multiprocessing by utilizing the multiprocess_ library
The `mpire.Worker` class is no longer directly available
```

--------------------------------

### Editable Install with Requirements

Source: https://github.com/sybrenjansen/mpire/blob/master/requirements.txt

This snippet demonstrates how to perform an editable installation of the current project, which is a common practice when developing Python packages. It ensures that changes made to the source code are immediately reflected without needing to reinstall.

```bash
pip install -e .
```

--------------------------------

### Install mpire using conda

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

This command installs the mpire package from the conda-forge channel using the conda package manager. This is an alternative installation method for users who prefer using conda.

```bash
conda install -c conda-forge mpire
```

--------------------------------

### Install MPIRE with Dill Support

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/install.rst

Installs MPIRE along with the 'dill' library, which provides more powerful serialization capabilities than the standard 'pickle'. This installation changes MPIRE's license to BSD.

```bash
pip install mpire[dill]
```

--------------------------------

### Start MPIRE Dashboard via Command Line

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/dashboard.rst

Starts the MPIRE dashboard using a bash script. This method is not available on Windows. Connection details are printed to the console. A custom port range can be specified using the --port-range argument.

```bash
$ mpire-dashboard

# Example with custom port range
$ mpire-dashboard --port-range 9000-9100
```

--------------------------------

### Manually Starting and Stopping WorkerPool

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/setup.rst

Illustrates the manual approach to starting and managing an mpire WorkerPool. It includes methods for stopping, joining, and terminating worker processes, with notes on potential deadlocks and cleanup.

```python
# Start a pool of 4 workers
pool = WorkerPool(n_jobs=4)

# Do some processing here
pass

# Only needed when keep_alive=True:
# Clean up pool (this will block until all processing has completed)
pool.stop_and_join()  # or use pool.join() which is an alias of stop_and_join()

# In the case you want to kill the processes, even though they are still busy
pool.terminate()

```

--------------------------------

### Install MPIRE via Conda

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/install.rst

Installs the MPIRE library using the conda package manager from the conda-forge channel. This is an alternative installation method, particularly useful for managing complex dependencies.

```bash
conda install -c conda-forge mpire
```

--------------------------------

### Basic Multiprocessing with MPIRE

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/index.rst

Demonstrates the core functionality of MPIRE using a simple map operation. This example showcases how to create a pool of workers and distribute tasks.

```python
from mpire import WorkerPool

def square(x):
    return x * x

if __name__ == "__main__":
    with WorkerPool(n_workers=4) as pool:
        results = pool.map(square, range(10))
    print(results)

```

--------------------------------

### AttributeError in IPython/Jupyter with MPIRE

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

Addresses the `AttributeError: Can't get attribute '<some_function>'` when using MPIRE in IPython or Jupyter notebooks, especially with the 'spawn' start method. Solutions include defining functions in importable files or using `use_dill=True`.

```python
import mpire

# If 'my_parallel_function' is defined in the notebook session:
# This can cause AttributeError with 'spawn' start method.

# Solution 1: Define the function in a separate Python file (e.g., 'my_module.py')
# from my_module import my_parallel_function
# pool = mpire.Pool()
# results = pool.map(my_parallel_function, data_list)

# Solution 2: Use dill
# pool = mpire.Pool(use_dill=True)
# results = pool.map(my_parallel_function, data_list)

```

--------------------------------

### Nested WorkerPools Configuration

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/setup.rst

Shows how to configure WorkerPools for nested usage, highlighting the 'daemon', 'start_method', and potential issues with different start methods like 'fork' and 'forkserver'.

```python
def job(...):
    with WorkerPool(n_jobs=4) as p:
        # Do some work
        results = p.map(...)

with WorkerPool(n_jobs=4, daemon=True, start_method='spawn') as pool:
    # This will raise an AssertionError telling you daemon processes
    # can't start child processes
    pool.map(job, ...)

with WorkerPool(n_jobs=4, daemon=False, start_method='spawn') as pool:
    # This will work just fine
    pool.map(job, ...)

```

--------------------------------

### Worker Initialization and Exit Example

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/map/worker_init_exit.rst

Demonstrates how to use worker_init to initialize worker state and worker_exit to clean up and return results. The example initializes a counter for even numbers within each worker and returns the count of even numbers processed by each worker.

```python
def init_func(worker_state):
    # Initialize a counter for each worker
    worker_state['count_even'] = 0

def square_and_count_even(worker_state, x):
    # Count number of even numbers and return the square
    if x % 2 == 0:
        worker_state['count_even'] += 1
    return x * x

def exit_func(worker_state):
    # Return the counter
    return worker_state['count_even']

with WorkerPool(n_jobs=4, use_worker_state=True) as pool:
    pool.map(square_and_count_even, range(100), worker_init=init_func, worker_exit=exit_func)
    print(pool.get_exit_results())  # Output, e.g.: [13, 13, 12, 12]
    print(sum(pool.get_exit_results()))  # Output: 50
```

--------------------------------

### mpire Dashboard Functions

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/reference/index.rst

Documentation for functions related to the mpire dashboard, including starting, connecting, and shutting down the dashboard, as well as managing stack levels for debugging.

```APIDOC
start_dashboard(port: int = 8080, host: str = '127.0.0.1', ...):
  Starts the mpire dashboard server.
  Parameters:
    port: The port to run the dashboard on.
    host: The host address for the dashboard.
    ...

connect_to_dashboard(port: int = 8080, host: str = '127.0.0.1', ...):
  Connects to an existing mpire dashboard server.
  Parameters:
    port: The port of the dashboard server.
    host: The host address of the dashboard server.
    ...

shutdown_dashboard(port: int = 8080, host: str = '127.0.0.1', ...):
  Shuts down the mpire dashboard server.
  Parameters:
    port: The port of the dashboard server.
    host: The host address of the dashboard server.
    ...

get_stacklevel() -> int:
  Gets the current stack level for the dashboard.
  Returns: The current stack level.

set_stacklevel(level: int):
  Sets the stack level for the dashboard.
  Parameters:
    level: The stack level to set.
```

--------------------------------

### mpire apply with worker init and exit

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/apply.rst

Shows how to use 'worker_init' and 'worker_exit' arguments with the 'apply' function to execute code when workers start and stop.

```python
from mpire import WorkerPool

def task(a):
    return a + 1

def worker_init():
    print("Worker started")

def worker_exit():
    print("Worker stopped")

with WorkerPool(n_jobs=5) as pool:
    pool.apply(task, 42, worker_init=worker_init, worker_exit=worker_exit)
```

--------------------------------

### Shared Objects with Copy-on-Write

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/getting_started.rst

Illustrates how to share objects between workers using mpire's 'shared_objects' option with the copy-on-write mechanism. Requires 'fork' start method and is not available on Windows.

```python
from mpire import WorkerPool

# Assume time_consuming_function is defined elsewhere
# def time_consuming_function(some_object, x):
#     time.sleep(1)
#     return some_object[x]

def main():
    some_object = {'data': list(range(10))}
    with WorkerPool(n_jobs=5, shared_objects=some_object, start_method='fork') as pool:
        results = pool.map(time_consuming_function, range(10), progress_bar=True)
```

--------------------------------

### Install ipywidgets for Jupyter Progress Bar

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

Installs the 'ipywidgets' package, which is required for the progress bar functionality in Jupyter notebooks. This can be done using either pip or conda.

```bash
pip install ipywidgets
```

```bash
conda install -c conda-forge ipywidgets
```

--------------------------------

### Enable Jupyter Widget Javascript Extension

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

Enables the Javascript extension for Jupyter widgets, which is necessary for them to function correctly in a notebook environment. This command should be run before starting the notebook server.

```bash
jupyter nbextension enable --py --sys-prefix widgetsnbextension
```

--------------------------------

### mpire.dashboard.start_dashboard and shutdown_dashboard

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/changelog.rst

Provides functionality to start and shut down the mpire dashboard. Includes a fix for freezing issues when ports are unavailable.

```python
mpire.dashboard.start_dashboard()
mpire.dashboard.shutdown_dashboard()
```

--------------------------------

### Start MPIRE Dashboard Programmatically

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/dashboard.rst

Starts the MPIRE dashboard and returns connection details. The dashboard attempts to use ports 8080-8100 by default, raising an OSError if none are available. A custom port range can be specified.

```python
from mpire.dashboard import start_dashboard

# Will return a dictionary with dashboard details
dashboard_details = start_dashboard()
print(dashboard_details)

# Example with custom port range
dashboard_details = start_dashboard(range(9000, 9100))
```

--------------------------------

### Enable and Use Worker Insights

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/worker_insights.rst

Demonstrates how to enable worker insights during the creation of a WorkerPool and how to retrieve the insights after tasks have been executed. It also shows an example of how to print the insights to the console.

```python
from mpire import WorkerPool
import time

def sleep_and_square(x):
    # For illustration purposes
    time.sleep(x / 1000)
    return x * x

with WorkerPool(n_jobs=4, enable_insights=True) as pool:
    pool.map(sleep_and_square, range(100))
    insights = pool.get_insights()
    print(insights)

# Example Output:
# {
#  'n_completed_tasks': [28, 24, 24, 24],
#  'total_start_up_time': '0:00:00.038',
#  'total_init_time': '0:00:00',
#  'total_waiting_time': '0:00:00.798',
#  'total_working_time': '0:00:04.980',
#  'total_exit_time': '0:00:00',
#  'total_time': '0:00:05.816',
#  'start_up_time': ['0:00:00.010', '0:00:00.008', '0:00:00.008', '0:00:00.011'],
#  'start_up_time_mean': '0:00:00.009',
#  'start_up_time_std': '0:00:00.001',
#  'start_up_ratio': 0.006610452621805033,
#  'init_time': ['0:00:00', '0:00:00', '0:00:00', '0:00:00'],
#  'init_time_mean': '0:00:00',
#  'init_time_std': '0:00:00',
#  'init_ratio': 0.0,
#  'waiting_time': ['0:00:00.309', '0:00:00.311', '0:00:00.165', '0:00:00.012'],
#  'waiting_time_mean': '0:00:00.199',
#  'waiting_time_std': '0:00:00.123',
#  'waiting_ratio': 0.13722942739284952,
#  'working_time': ['0:00:01.142', '0:00:01.135', '0:00:01.278', '0:00:01.423'],
#  'working_time_mean': '0:00:01.245',
#  'working_time_std': '0:00:00.117',
#  'working_ratio': 0.8561601182661567,
#  'exit_time': ['0:00:00', '0:00:00', '0:00:00', '0:00:00'],
#  'exit_time_mean': '0:00:00',
#  'exit_time_std': '0:00:00',
#  'exit_ratio': 0.0,
#  'top_5_max_task_durations': ['0:00:00.099', '0:00:00.098', '0:00:00.097', '0:00:00.096', '0:00:00.095'],
#  'top_5_max_task_args': ['Arg 0: 99', 'Arg 0: 98', 'Arg 0: 97', 'Arg 0: 96', 'Arg 0: 95']
# }

```

--------------------------------

### MPIRE Shared Objects (Copy-on-Write)

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/index.rst

Demonstrates the use of copy-on-write shared objects with MPIRE. This feature, available with the 'fork' start method, allows workers to efficiently access shared data without explicit synchronization.

```python
from mpire import WorkerPool
from multiprocessing import Manager

if __name__ == "__main__":
    with Manager() as manager:
        shared_list = manager.list([1, 2, 3])
        
        def modify_shared(item):
            shared_list.append(item)
            return len(shared_list)

        with WorkerPool(n_workers=2) as pool:
            results = pool.map(modify_shared, range(5))
        
        print("Final shared list:", list(shared_list))
        print("Results:", results)

```

--------------------------------

### WorkerPool CPU Pinning Examples

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/cpu_pinning.rst

Demonstrates various ways to pin mpire worker processes to specific CPU IDs using the `cpu_ids` parameter in the `WorkerPool` constructor. This includes pinning to single cores, ranges of cores, and assigning multiple cores to each process. CPU IDs must be positive integers and not exceed the available CPU count. Pinning is not supported on macOS or when using threading.

```python
from mpire import WorkerPool

# Pin the two child processes to CPUs 2 and 3
with WorkerPool(n_jobs=2, cpu_ids=[2, 3]) as pool:
    pass

# Pin the child processes to CPUs 40-59
with WorkerPool(n_jobs=20, cpu_ids=list(range(40, 60))) as pool:
    pass

# All child processes have to share a single core:
with WorkerPool(n_jobs=4, cpu_ids=[0]) as pool:
    pass

# All child processes have to share multiple cores, namely 4-7:
with WorkerPool(n_jobs=4, cpu_ids=[[4, 5, 6, 7]]) as pool:
    pass

# Each child process can use two distinctive cores:
with WorkerPool(n_jobs=4, cpu_ids=[[0, 1], [2, 3], [4, 5], [6, 7]]) as pool:
    pass

# Disable CPU pinning (default behavior)
with WorkerPool(n_jobs=4, cpu_ids=None) as pool:
    pass
```

--------------------------------

### Jinja2 Template - Task Management Table

Source: https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/templates/index.html

This Jinja2 template snippet includes a Jinja2 file named 'mpire.html', which presumably contains the structure for displaying task progress, duration, remaining time, start, and finish/ETA information.

```jinja2
{% include 'mpire.html' %}
```

--------------------------------

### Build mpire Documentation

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Provides instructions for building the mpire documentation. This can be done using setup.py for older Python versions or directly from the docs folder using make html.

```bash
python setup.py build_docs
```

```bash
make html
```

--------------------------------

### Worker Initialization and Task Execution

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/getting_started.rst

Demonstrates how to initialize worker state before tasks begin using 'worker_init' and access this state within tasks. Also mentions 'worker_exit' for cleanup.

```python
from mpire import WorkerPool

# Assume task and init functions are defined elsewhere
# def init(worker_state):
#     worker_state['dataset'] = list(range(100))
#     worker_state['model'] = 'dummy_model'

# def task(worker_state, idx):
#     return f"Processed {idx} with model {worker_state['model']} on data {worker_state['dataset'][idx]}"

with WorkerPool(n_jobs=5, use_worker_state=True) as pool:
    results = pool.map(task, range(10), worker_init=init)
```

--------------------------------

### Worker Insights and get_insights

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Demonstrates how to enable and retrieve worker insights to analyze performance metrics like startup time, waiting time, and working time. This helps in diagnosing performance bottlenecks.

```python
from mpire import WorkerPool
import time

def time_consuming_function(x):
    time.sleep(0.1)
    return x

with WorkerPool(n_jobs=5, enable_insights=True) as pool:
    results = pool.map(time_consuming_function, range(10))
    insights = pool.get_insights()
```

--------------------------------

### Enabling Worker Insights

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/getting_started.rst

Shows how to enable worker insights to profile worker startup, waiting, and working times by setting 'enable_insights' to True in WorkerPool.

```python
from mpire import WorkerPool

# Assume time_consuming_function is defined elsewhere
# def time_consuming_function(x):
#     time.sleep(1)
#     return x * 2

with WorkerPool(n_jobs=5, enable_insights=True) as pool:
    results = pool.map(time_consuming_function, range(10))

# insights = pool.get_insights()
```

--------------------------------

### Worker Initialization and Task Execution

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Shows how to initialize workers with custom state, such as loading models or datasets, using the `worker_init` parameter. The `worker_state` is then accessible within each task.

```python
from mpire import WorkerPool
import time

def init(worker_state):
    # Load a big dataset or model and store it in a worker specific worker_state
    worker_state['dataset'] = list(range(100))
    worker_state['model'] = lambda x: x * 2 # Dummy model

def task(worker_state, idx):
    # Let the model predict a specific instance of the dataset
    return worker_state['model'](worker_state['dataset'][idx])

with WorkerPool(n_jobs=5, use_worker_state=True) as pool:
    results = pool.map(task, range(10), worker_init=init)
```

--------------------------------

### Git Tagging for Release

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/contributing.rst

This command is used to create an annotated tag for a specific release version. It requires the version number and a commit message.

```bash
git tag -a vX.Y.Z -m "vX.Y.Z"
```

--------------------------------

### Basic Parallel Execution with mpire

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/getting_started.rst

Demonstrates how to use mpire's WorkerPool as a drop-in replacement for multiprocessing.Pool to parallelize a time-consuming function.

```python
from mpire import WorkerPool

# Assume time_consuming_function is defined elsewhere
# def time_consuming_function(x):
#     time.sleep(1)
#     return x * 2

with WorkerPool(n_jobs=5) as pool:
    results = pool.map(time_consuming_function, range(10))
```

--------------------------------

### mpire Worker Pool with Nested Pools

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/changelog.rst

Version 0.4.0 enabled the creation of nested `mpire.WorkerPool` instances by allowing workers to be started as normal (non-daemon) child processes.

```python
from mpire import WorkerPool

def inner_task(x):
    return x * 2

def outer_task(data):
    with WorkerPool(n_jobs=2) as inner_pool:
        return inner_pool.map(inner_task, data)

with WorkerPool(n_jobs=2) as outer_pool:
    results = outer_pool.map(outer_task, [[1, 2], [3, 4]])

# results: [[2, 4], [6, 8]]
```

--------------------------------

### Running Unittests with Python -m unittest

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

An alternative method for running unittests that avoids issues with the multiprocessing semaphore tracker. This command is recommended over 'python setup.py test -s tests.some_test' when encountering KeyError or UserWarning related to semaphores.

```bash
python -m unittest tests.some_test
```

--------------------------------

### mpire WorkerPool apply_async deadlock prevention

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/apply.rst

Illustrates how to avoid deadlocks when using 'apply_async' by properly managing the pool lifecycle, for example, using 'stop_and_join()'.

```python
from mpire import WorkerPool

def task(a, b):
    return a + b

with WorkerPool(n_jobs=4) as pool:
    async_results = [pool.apply_async(task, args=(i, i)) for i in range(10)]
    pool.stop_and_join()

    # Will not deadlock
    results = [async_result.get() for async_result in async_results]
```

--------------------------------

### Windows Specific Troubleshooting for MPIRE

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/troubleshooting.rst

Provides notes for Windows users of MPIRE, mentioning that `OSError` messages during exceptions with `dill` can be ignored and that the `mpire-dashboard` script is not supported on Windows.

```text
* When using ``dill`` and an exception occurs, or when the exception occurs in an exit function, it can print additional ``OSError`` messages in the terminal, but they can be safely ignored.
* The ``mpire-dashboard`` script does not work on Windows.
```

--------------------------------

### Performance Insights - Time Distribution

Source: https://github.com/sybrenjansen/mpire/blob/master/mpire/dashboard/templates/progress_bar.html

Details the various time components involved in worker processes, such as startup, initialization, waiting, working, and exit times. This breakdown is crucial for identifying performance bottlenecks.

```python
ⓘ Start up time denotes the time to spin up a worker. Init time is the time a worker spends on the initialization function, when provided. Waiting time is the time a worker needs to wait for new tasks to come in. Working time is the time a worker spends on the task at hand. Exit time is the time a worker spends on the exit function, when provided.
```

--------------------------------

### mpire AsyncResult methods

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/apply.rst

Shows how to interact with the AsyncResult object returned by 'apply_async', including checking completion status, waiting, getting results, and checking success.

```python
from mpire import WorkerPool

def task(a, b):
    return a + b

with WorkerPool(n_jobs=1) as pool:
    async_result = pool.apply_async(task, args=(1, 1))

    # Check if the task is completed
    is_completed = async_result.ready()

    # Wait until the task is completed, or until the timeout is reached.
    async_result.wait(timeout=10)

    # Get the result of the task. This will block until the task is completed,
    # or until the timeout is reached.
    result = async_result.get(timeout=None)

    # Check if the task was successful (i.e., did not raise an exception).
    # This will raise an exception if the task is not completed yet.
    is_successful = async_result.successful()
```

--------------------------------

### mpire.WorkerPool.map with NumPy Array Concatenation

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/changelog.rst

Starting from version 0.8.0, the `mpire.WorkerPool.map` method supports automatic concatenation of NumPy array outputs. This simplifies working with parallelized NumPy operations by directly combining results.

```python
from mpire import WorkerPool

def process_array(x):
    # Example function that returns a numpy array
    import numpy as np
    return np.array([x, x*2])

with WorkerPool(n_jobs=2) as pool:
    results = pool.map(process_array, range(5))

# results will be a single concatenated numpy array
# e.g., [[0, 0], [1, 2], [2, 4], [3, 6], [4, 8]]
```

--------------------------------

### Parallel Function Execution with mpire

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/getting_started.rst

Demonstrates how to use `pool.map` to execute a time-consuming function across multiple processes and retrieve insights about the workers. This is useful for distributing CPU-bound tasks.

```python
results = pool.map(time_consuming_function, range(10))
insights = pool.get_insights()
```

--------------------------------

### Using multiprocessing.Array as Shared Objects

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/shared_objects.rst

Shows how to use `multiprocessing.Array` as shared objects when copy-on-write is not available. This example demonstrates storing results from multiple tasks into shared arrays, with locking disabled for performance where applicable.

```python
from multiprocessing import Array
from mpire import WorkerPool

def square_add_and_modulo_with_index(shared_objects, idx, x):
    # Unpack results containers
    square_results_container, add_results_container = shared_objects

    # Square, add and modulo
    square_results_container[idx] = x * x
    add_results_container[idx] = x + x
    return x % 2

def main():
    # Use a shared array of size 100 and type float to store the results
    square_results_container = Array('f', 100, lock=False)
    add_results_container = Array('f', 100, lock=False)
    shared_objects = square_results_container, add_results_container
    with WorkerPool(n_jobs=4, shared_objects=shared_objects) as pool:

        # Square, add and modulo the results and store them in the results containers
        modulo_results = pool.map(square_add_and_modulo_with_index,
                                  enumerate(range(100)), iterable_len=100)
```

--------------------------------

### Basic Map Usage with Single Argument

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/map/map.rst

Demonstrates the basic usage of the `pool.map` function with a simple function that squares numbers. It shows how to initialize the WorkerPool and process an iterable of arguments.

```python
from mpire import WorkerPool

def square(x):
    return x * x

with WorkerPool(n_jobs=4) as pool:
    # 1. Square the numbers, results should be: [0, 1, 4, 9, 16, 25, ...]
    results = pool.map(square, range(100))
```

--------------------------------

### Using Shared Objects with WorkerPool (fork)

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/shared_objects.rst

Demonstrates how to pass shared objects to worker processes using the `shared_objects` argument in the `WorkerPool` constructor when using the 'fork' start method. Shared objects are treated as copy-on-write.

```python
from mpire import WorkerPool

def task(dataset, x):
    # Do something with this copy-on-write dataset
    ...

def main():
    dataset = ... # Load big dataset
    with WorkerPool(n_jobs=4, shared_objects=dataset, start_method='fork') as pool:
        ... = pool.map(task, range(100))
```

--------------------------------

### mpire CPU Pinning

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/changelog.rst

Starting from version 0.5.0, mpire supports CPU pinning, allowing child processes to be assigned to specific CPUs or a range of CPUs. This can improve performance by reducing context switching and cache misses.

```python
from mpire import WorkerPool

def cpu_bound_task(x):
    # Simulate CPU-intensive work
    return sum(i*i for i in range(x))

# Pin workers to CPU cores 0 and 1
with WorkerPool(n_jobs=2, cpu_affinity=[0, 1]) as pool:
    results = pool.map(cpu_bound_task, range(1000))

# Pin workers to a range of CPU cores (e.g., 2 to 3)
with WorkerPool(n_jobs=2, cpu_affinity=[2, 3]) as pool:
    results = pool.map(cpu_bound_task, range(1000))

```

--------------------------------

### Progress Bar with WorkerPool

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Demonstrates how to enable a progress bar for tasks processed by the WorkerPool. This provides visual feedback on task completion status.

```python
from mpire import WorkerPool
import time

def time_consuming_function(x):
    time.sleep(1)
    return x * 2

with WorkerPool(n_jobs=5) as pool:
    results = pool.map(time_consuming_function, range(10), progress_bar=True)
```

--------------------------------

### Shared Objects with WorkerPool

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

Illustrates the use of `shared_objects` to efficiently share data across worker processes. This is particularly useful for large objects, leveraging copy-on-write semantics when the 'fork' start method is used.

```python
from mpire import WorkerPool
import time

def time_consuming_function(some_object, x):
    time.sleep(1)
    # Access or modify some_object here
    return x

def main():
    some_object = {'data': list(range(1000))}
    with WorkerPool(n_jobs=5, shared_objects=some_object) as pool:
        results = pool.map(time_consuming_function, range(10), progress_bar=True)

main()
```

--------------------------------

### MPIRE Progress Bar Integration

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/index.rst

Shows how to integrate the `tqdm` library with MPIRE to display a progress bar for long-running tasks. This provides visual feedback on the multiprocessing progress.

```python
import time
from mpire import WorkerPool

def task_with_progress(x):
    time.sleep(0.1)
    return x * x

if __name__ == "__main__":
    with WorkerPool(n_workers=4, progress_bar=True) as pool:
        results = pool.map(task_with_progress, range(20))
    print(results)

```

--------------------------------

### Enable Basic Progress Bar

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/map/progress_bar.rst

Enables a basic tqdm progress bar for the map function, showing elapsed time, completion percentage, and task speed. Requires the tqdm package.

```python
from mpire import WorkerPool

# Assuming 'task' is a defined function and range(100) is the iterable
with WorkerPool(n_jobs=4) as pool:
    pool.map(task, range(100), progress_bar=True)
```

--------------------------------

### Elaborate Example: Worker ID with Shared Array

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/worker_id.rst

Illustrates an advanced use case where the worker ID is used to access specific elements in a shared array, avoiding the need for locks. Each worker squares a number and adds it to its designated slot in the `results_container`.

```python
from multiprocessing import Array
from mpire import WorkerPool

def square_sum(worker_id, shared_objects, x):
    # Even though the shared objects is a single container, we 'unpack' it anyway
    results_container = shared_objects

    # Square and sum
    results_container[worker_id] += x * x

# Use a shared array of size equal to the number of jobs to store the results
results_container = Array('f', 4, lock=False)

with WorkerPool(n_jobs=4, shared_objects=results_container, pass_worker_id=True) as pool:
    # Square the results and store them in the results container
    pool.map_unordered(square_sum, range(100))
```

--------------------------------

### Keep Alive with Changing worker_init and worker_exit

Source: https://github.com/sybrenjansen/mpire/blob/master/docs/usage/workerpool/keep_alive.rst

Shows an example where `keep_alive` is enabled, and `worker_init` and `worker_exit` functions are changed between `map` calls. It highlights that `worker_init` is only called on worker startup and `worker_exit` on termination, leading to potential side effects if not managed carefully.

```python
from mpire import WorkerPool

def init_func_1(): pass
def exit_func_1(): pass

def init_func_2(): pass
def exit_func_2(): pass

def task(x):
    pass

with WorkerPool(n_jobs=4, keep_alive=True) as pool:
    pool.map(task, range(100), worker_init=init_func_1, worker_exit=exit_func_1)
    pool.map(task, range(100), worker_init=init_func_2, worker_exit=exit_func_2)
```

--------------------------------

### Basic multiprocessing with mpire.WorkerPool

Source: https://github.com/sybrenjansen/mpire/blob/master/README.rst

This Python code snippet shows how to use mpire's `WorkerPool` as a replacement for `multiprocessing.Pool`. It demonstrates the similar syntax for parallelizing a function using `pool.map`.

```python
from mpire import WorkerPool

with WorkerPool(n_jobs=5) as pool:
    results = pool.map(time_consuming_function, range(10))
```