Python Multiprocessing

https://superfastpython.com/multiprocessing-in-python/
Admin
Python Multiprocessing provides process-based parallelism in Python, enabling the use of multiple...
Tokens:35,591
Snippets:321
Trust Score:9.5
Update:9 months ago
Show doc for...
Tokens:
Raw
### Get All Supported Multiprocessing Start Methods in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves a list of all start methods supported by the multiprocessing module on the current platform. This is useful for understanding available options before setting a start method.

```python
import multiprocessing

# get supported start methods
methods = multiprocessing.get_all_start_methods()
```

--------------------------------

### Complete Example: Run Function in New Process

Source: https://superfastpython.com/multiprocessing-in-python/index

This comprehensive Python example shows how to define a function, create a `multiprocessing.Process` to run it, start the process, and wait for its completion using `join()`. It includes necessary imports and the standard `if __name__ == '__main__':` block.

```python
# SuperFastPython.com
# example of running a function in another process
from time import sleep
from multiprocessing import Process

# a custom function that blocks for a moment
def task():
    # block for a moment
    sleep(1)
    # display a message
    print('This is from another process')

# entry point
if __name__ == '__main__':
    # create a process
    process = Process(target=task)
    # run the process
    process.start()
    # wait for the process to finish
    print('Waiting for the process...')
    process.join()
```

--------------------------------

### Complete Example: Extending Process Class in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

This is a complete Python example demonstrating how to extend the `multiprocessing.Process` class. It defines a `CustomProcess` with an overridden `run` method, then creates, starts, and joins an instance of this process. The example highlights the lifecycle of a child process.

```python
# SuperFastPython.com
# example of extending the Process class
from time import sleep
from multiprocessing import Process

# custom process class
class CustomProcess(Process):
    # override the run function
    def run(self):
        # block for a moment
        sleep(1)
        # display a message
        print('This is coming from another process')

# entry point
if __name__ == '__main__':
    # create the process
    process = CustomProcess()
    # start the process
    process.start()
    # wait for the process to finish
    print('Waiting for the process to finish')
    process.join()
```

--------------------------------

### Instantiate and Start Custom Process

Source: https://superfastpython.com/multiprocessing-in-python/index

This code demonstrates how to create an instance of a custom `Process` subclass (e.g., `CustomProcess`) and initiate its execution using the `start()` method. The `start()` method internally calls the overridden `run()` method in a new process. This is a core step in multiprocessing.

```python
# create the process
process = CustomProcess()
# start the process
process.start()
```

--------------------------------

### Create and Start a Basic Process

Source: https://superfastpython.com/multiprocessing-in-python/index

This snippet demonstrates the fundamental steps to run a function in a new process. It involves creating a `multiprocessing.Process` object, specifying the target function, and then calling `start()` to initiate execution. The `start()` method returns immediately, allowing the OS to schedule the function in a separate process.

```python
# create a process
process = multiprocessing.Process(target=task)
```

--------------------------------

### Start a New Process

Source: https://superfastpython.com/multiprocessing-in-python/index

This snippet shows how to start the execution of a previously created `multiprocessing.Process` object. Calling `start()` allows the operating system to schedule and run the target function in a separate process as soon as possible.

```python
# run the new process
process.start()
```

--------------------------------

### Complete Example: Process Lifecycle and Restart Attempt in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

A full Python script demonstrating the creation, starting, joining, and subsequent failed restart attempt of a multiprocessing process. It includes the necessary imports and the main execution block, showcasing the `AssertionError`.

```python
# SuperFastPython.com
# example of restarting a process
from time import sleep
from multiprocessing import Process

# custom target function
def task():
    # block for a moment
    sleep(1)
    # report a message
    print('Hello from the new process')

# entry point
if __name__ == '__main__':
    # create a new process
    process = Process(target=task)

    # start the process
    process.start()

    # wait for the process to finish
    process.join()

    # try and start the process again
    process.start()
```

--------------------------------

### Worked Example: Safely Stopping a Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

A complete Python example demonstrating the safe termination of a child process using `multiprocessing.Event`. It includes creating the event, starting a process that checks the event, setting the event from the parent process, and joining the child process.

```python
# SuperFastPython.com
# example of stopping a new process
from time import sleep
from multiprocessing import Process
from multiprocessing import Event

# custom task function
def task(event):
    # execute a task in a loop
    for i in range(5):
        # block for a moment
        sleep(1)
        # check for stop
        if event.is_set():
            break
        # report a message
        print('Worker process running...', flush=True)
    print('Worker closing down', flush=True)

# entry point
if __name__ == '__main__':
    # create the event
    event = Event()
    # create and configure a new process
    process = Process(target=task, args=(event,))
    # start the new process
    process.start()
    # block for a while
    sleep(3)
    # stop the worker process
    print('Main stopping process')
    event.set()
    # wait for the new process to finish
    process.join()
```

--------------------------------

### Create and Start a Multiprocessing Process in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates the creation of a new process instance using `multiprocessing.Process` and configuring it to run a specific target function. It then starts the execution of this new process.

```python
...# create a new process
process = Process(target=task)

...# start the process
process.start()
```

--------------------------------

### Get Current Multiprocessing Start Method in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves the currently configured start method for the multiprocessing module. This helps in understanding the active process creation strategy.

```python
import multiprocessing

# get the current start method
method = multiprocessing.get_start_method()
```

--------------------------------

### Complete Semaphore Example in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

A complete Python script demonstrating the use of multiprocessing.Semaphore. It includes the task function, semaphore creation, process instantiation, starting, and joining, to limit concurrent execution to two processes at a time.

```python
# SuperFastPython.com
# example of using a semaphore
from time import sleep
from random import random
from multiprocessing import Process
from multiprocessing import Semaphore

# target function
def task(semaphore, number):
    # attempt to acquire the semaphore
    with semaphore:
        # simulate computational effort
        value = random()
        sleep(value)
        # report result
        print(f'Process {number} got {value}')

# entry point
if __name__ == '__main__':
    # create the shared semaphore
    semaphore = Semaphore(2)

    # create processes
    processes = [Process(target=task, args=(semaphore, i)) for i in range(10)]

    # start child processes
    for process in processes:
        process.start()

    # wait for child processes to finish
    for process in processes:
        process.join()
```

--------------------------------

### Force Set Multiprocessing Start Method via Context in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to forcefully set the start method on a multiprocessing context, even if it has already been configured. This is an advanced option and should be used with caution.

```python
import multiprocessing

# get a context configured with a start method
context = multiprocessing.get_context('fork')

# set the start method
context.set_start_method('spawn', force=True)
```

--------------------------------

### Set Multiprocessing Start Method in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Explains how to set the start method for child processes using multiprocessing.set_start_method(). The method must be one of the available options for the platform.

```python
import multiprocessing

multiprocessing.set_start_method('spawn')
```

--------------------------------

### Complete Python Example for Multiprocessing Barrier

Source: https://superfastpython.com/multiprocessing-in-python/index

This comprehensive Python script integrates all components for demonstrating `multiprocessing.Barrier`. It defines the `task` function, initializes the barrier, creates and starts worker processes, and includes the main process waiting logic. The script ensures proper synchronization before reporting the final completion message.

```python
# SuperFastPython.com
# example of using a barrier with processes
from time import sleep
from random import random
from multiprocessing import Process
from multiprocessing import Barrier

# target function to prepare some work
def task(barrier, number):
    # generate a unique value
    value = random() * 10
    # block for a moment
    sleep(value)
    # report result
    print(f'Process {number} done, got: {value}', flush=True)
    # wait on all other processes to complete
    barrier.wait()

# entry point
if __name__ == '__main__':
    # create a barrier
    barrier = Barrier(5 + 1)

    # create the worker processes
    for i in range(5):
        # start a new process to perform some work
        worker = Process(target=task, args=(barrier, i))
        worker.start()

    # wait for all processes to finish
    print('Main process waiting on all results...')
    barrier.wait()

    # report once all processes are done
    print('All processes have their result')
```

--------------------------------

### Full Example: Custom Process with Shared Value and Parent Access (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

This is the complete Python script demonstrating how to extend `multiprocessing.Process` to use shared attributes. It defines a `CustomProcess` with a shared `Value`, starts it, waits for it to finish, and then accesses the updated shared value from the parent process.

```python
# SuperFastPython.com
# example of extending the Process class and adding shared attributes
from time import sleep
from multiprocessing import Process
from multiprocessing import Value

# custom process class
class CustomProcess(Process):
    # override the constructor
    def __init__(self):
        # execute the base constructor
        Process.__init__(self)
        # initialize integer attribute
        self.data = Value('i', 0)

    # override the run function
    def run(self):
        # block for a moment
        sleep(1)
        # store the data variable
        self.data.value = 99
        # report stored value
        print(f'Child stored: {self.data.value}')

# entry point
if __name__ == '__main__':
    # create the process
    process = CustomProcess()
    # start the process
    process.start()
    # wait for the process to finish
    print('Waiting for the child process to finish')
    # block until child process is terminated
    process.join()
    # report the process attribute
    print(f'Parent got: {process.data.value}')
```

--------------------------------

### Python Multiprocessing Event Example

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates coordinating multiple child processes using a `multiprocessing.Event`. Child processes wait for the event to be set by the main process before executing their tasks. This ensures synchronized execution start.

```python
# SuperFastPython.com
# example of using an event object with processes
from time import sleep
from random import random
from multiprocessing import Process
from multiprocessing import Event

# target task function
def task(event, number):
    # wait for the event to be set
    print(f'Process {number} waiting...', flush=True)
    event.wait()
    # begin processing
    value = random()
    sleep(value)
    print(f'Process {number} got {value}', flush=True)

# entry point
if __name__ == '__main__':
    # create a shared event object
    event = Event()
    # create a suite of processes
    processes = [Process(target=task, args=(event, i)) for i in range(5)]
    # start all processes
    for process in processes:
        process.start()
    # block for a moment
    print('Main process blocking...')
    sleep(2)
    # trigger all child processes
    event.set()
    # wait for all child processes to terminate
    for process in processes:
        process.join()
```

--------------------------------

### Python: Demonstrate Reentrant Lock with Processes

Source: https://superfastpython.com/multiprocessing-in-python/index

This example demonstrates using `multiprocessing.RLock` to protect shared resources between processes. It defines functions for reporting and performing tasks, creates multiple processes that utilize the lock, and ensures proper synchronization by starting and joining processes. The reentrant nature of `RLock` prevents deadlocks when a process holding the lock needs to re-acquire it.

```python
# SuperFastPython.com
# example of a reentrant lock for processes
from time import sleep
from random import random
from multiprocessing import Process
from multiprocessing import RLock

# reporting function
def report(lock, identifier):
    # acquire the lock with lock:
    print(f'>process {identifier} done')

# work function
def task(lock, identifier, value):
    # acquire the lock with lock:
    print(f'>process {identifier} sleeping for {value}')
    sleep(value)
    # report
    report(lock, identifier)

# entry point
if __name__ == '__main__':
    # create a shared reentrant lock
    lock = RLock()

    # create processes
    processes = [Process(target=task, args=(lock, i, random())) for i in range(10)]

    # start child processes
    for process in processes:
        process.start()

    # wait for child processes to finish
    for process in processes:
        process.join()

```

--------------------------------

### Configure Process as Daemon via Property

Source: https://superfastpython.com/multiprocessing-in-python/index

This example shows how to create a standard multiprocessing.Process instance and then subsequently configure it to be a daemon process by setting its 'daemon' property to True. Finally, it reports the daemon status.

```python
# SuperFastPython.com
# example of setting a process to be a daemon via the property
from multiprocessing import Process

# entry point
if __name__ == '__main__':
    # create a process
    process = Process()
    # configure the process to be a daemon
    process.daemon = True
    # report if the process is a daemon
    print(process.daemon)
```

--------------------------------

### Set Multiprocessing Start Method within Protected Entry Point in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates the best practice of setting the multiprocessing start method within a protected entry point (`if __name__ == '__main__'`) to avoid RuntimeErrors.

```python
import multiprocessing

if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')
```

--------------------------------

### Get Multiprocessing Context by Start Method in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves a multiprocessing context object configured with a specific start method. This context can then be used to create new child processes.

```python
import multiprocessing

# get a context configured with a start method
context = multiprocessing.get_context('fork')
```

--------------------------------

### Create Daemon Process via Constructor

Source: https://superfastpython.com/multiprocessing-in-python/index

This example demonstrates creating a new process and configuring it as a daemon directly within the multiprocessing.Process constructor by setting the 'daemon' argument to True. It then prints the daemon status of the created process.

```python
# SuperFastPython.com
# example of setting a process to be a daemon via the constructor
from multiprocessing import Process

# entry point
if __name__ == '__main__':
    # create a daemon process
    process = Process(daemon=True)
    # report if the process is a daemon
    print(process.daemon)
```

--------------------------------

### Create and Start a Process with Arguments

Source: https://superfastpython.com/multiprocessing-in-python/index

This code illustrates how to pass arguments to a function executed in a new process. Arguments can be provided as a tuple to the `args` parameter or as a dictionary to the `kwargs` parameter of the `multiprocessing.Process` constructor.

```python
# create a process
process = multiprocessing.Process(target=task, args=(arg1, arg2))
```

--------------------------------

### Log Messages Directly (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

This example demonstrates logging an informational message directly after configuring the logger. While simple, this method is not recommended for multiprocessing due to potential race conditions and data corruption when multiple processes write to the same log target.

```python
# report a message
logging.info('Hello world.')
```

--------------------------------

### Set Multiprocessing Start Method in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Sets the start method for creating child processes. It's crucial to call this before any other multiprocessing code and within a protected entry point (`if __name__ == '__main__':`).

```python
import multiprocessing

if __name__ == '__main__':
    # set the start method
    multiprocessing.set_start_method('spawn')
```

--------------------------------

### Create and Start Processes with Semaphore

Source: https://superfastpython.com/multiprocessing-in-python/index

Creates a multiprocessing.Semaphore with a limit of 2 concurrent processes. Then, it generates a list of 10 child processes, each targeting the 'task' function and passing the shared semaphore and a unique identifier. Finally, it starts all created processes.

```python
# create the shared semaphoresemaphore = Semaphore(2)

# create processes
processes = [Process(target=task, args=(semaphore, i)) for i in range(10)]

# start child processes
for process in processes:
    process.start()
```

--------------------------------

### Create Child Process Using a Multiprocessing Context in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to create a child process using a multiprocessing context object obtained via `get_context()`. This allows for process creation with a pre-defined start method.

```python
import multiprocessing

# get a context configured with a start method
context = multiprocessing.get_context('fork')

# create a child process via a context
process = context.Process(...)
```

--------------------------------

### Protect Entry Point for Multiprocessing in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

This code snippet demonstrates the essential idiom for protecting the entry point of a Python script when using the multiprocessing module. It ensures that new processes are only started when the script is run directly, preventing 'RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.' This is crucial for platforms like Windows and macOS that use the 'spawn' start method.

```python
if __name__ == '__main__':
    # Code to start new processes goes here
    pass
```

--------------------------------

### Create and Start Worker Processes with a Barrier

Source: https://superfastpython.com/multiprocessing-in-python/index

This Python code snippet demonstrates the creation and initiation of multiple worker processes. It initializes a `multiprocessing.Barrier` object, specifying the total number of parties (including the main process) that must reach the barrier. Then, it iterates to create and start several child processes, each targeting the `task` function and passing the barrier and a unique ID.

```python
# create a barrier
barrier = Barrier(5 + 1)

# create the worker processes
for i in range(5):
    # start a new process to perform some work
    worker = Process(target=task, args=(barrier, i))
    worker.start()
```

--------------------------------

### Main Module Idiom for Multiprocessing (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

This example combines protecting the entry point and adding freeze support, known as the 'main module' idiom. It ensures that multiprocessing works correctly both in standard Python execution and when the program is frozen into an executable, preventing common `RuntimeError` issues.

```python
import multiprocessing

if __name__ == '__main__':
    multiprocessing.freeze_support()
    # ... your multiprocessing code here ...
```

--------------------------------

### Python: Main Process Waiting on Condition Variable

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates the main process acquiring a `Condition` variable, starting a child process, and then waiting for a notification. It highlights the importance of acquiring the lock before waiting to prevent race conditions.

```python
# SuperFastPython.com
# example of wait/notify with a condition for processes
from time import sleep
from multiprocessing import Process
from multiprocessing import Condition

# target function to prepare some work
def task(condition):
    # block for a moment
    sleep(1)
    # notify a waiting process that the work is done
    print('Child process sending notification...', flush=True)
    with condition:
        condition.notify()
    # do something else...
    sleep(1)

# entry point
if __name__ == '__main__':
    # create a condition
    condition = Condition()

    # wait to be notified that the data is ready
    print('Main process waiting for data...')
    with condition:
        # start a new process to perform some work
        worker = Process(target=task, args=(condition,))
        worker.start()
        # wait to be notified
        condition.wait()

    # we know the data is ready
    print('Main process all done')
```

--------------------------------

### Set Multiprocessing Start Method in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

This code snippet shows how to explicitly set the start method for Python's multiprocessing module. It is particularly useful for ensuring consistent behavior across different operating systems or when needing to use a specific start method like 'fork' (though 'fork' is not available on Windows). This can help mitigate issues related to buffering and process initialization.

```python
import multiprocessing

# Set the start method to 'fork' (if available and desired)
# multiprocessing.set_start_method('fork')

# Set the start method to 'spawn' (default on Windows/macOS)
# multiprocessing.set_start_method('spawn')

# Set the start method to 'forkserver' (if available and desired)
# multiprocessing.set_start_method('forkserver')
```

--------------------------------

### Attempt to Restart a Terminated Process in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates the consequence of attempting to call `start()` on a process that has already terminated. This action will result in an `AssertionError`, highlighting that processes cannot be reused.

```python
...# try and start the process again
process.start()
```

--------------------------------

### Get Main Process Instance from Main Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to retrieve the `multiprocessing.Process` instance for the main process when executing code within it. This is achieved using the `multiprocessing.current_process()` function.

```python
# SuperFastPython.com
# example of getting the main process from the main process
from multiprocessing import current_process

# get the process instance
process = current_process()

# report details of the main process
print(process)
```

--------------------------------

### Protect Entry Point for Multiprocessing

Source: https://superfastpython.com/multiprocessing-in-python/index

Safeguards the code that starts new processes by checking if the script is the main entry point using `if __name__ == '__main__':`. This prevents `RuntimeError` on platforms like Windows and macOS that use the 'spawn' start method, ensuring child processes are only created when the script is executed directly.

```python
import multiprocessing

def worker_function():
    print("Worker process started.")

if __name__ == '__main__':
    # This code will only run when the script is executed directly
    print("Main process started.")
    p = multiprocessing.Process(target=worker_function)
    p.start()
    p.join()
    print("Main process finished.")
```

--------------------------------

### Protect Entry Point for Multiprocessing (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

This code snippet demonstrates how to protect the entry point of a Python program using the `if __name__ == '__main__':` idiom. This is crucial for preventing `RuntimeError` when creating child processes, especially with the 'spawn' start method used by default on Windows and macOS.

```python
if __name__ == '__main__':
    # ... your multiprocessing code here ...
```

--------------------------------

### Get Size of Multiprocessing Queue in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to check the current number of items in a multiprocessing.Queue using the qsize() method. This returns the approximate size.

```python
import multiprocessing

queue = multiprocessing.Queue()
queue.put("item1")
queue.put("item2")

# check the size of the queue
size = queue.qsize()
```

--------------------------------

### Run Function with Arguments in New Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to execute a custom function with arguments in a separate process. The 'task' function takes 'sleep_time' and 'message' as arguments. The multiprocessing.Process constructor is used with the 'args' parameter to pass these arguments as a tuple. The process is started and then joined to wait for its completion.

```python
# SuperFastPython.com
# example of running a function with arguments in another process
from time import sleep
from multiprocessing import Process

# a custom function that blocks for a moment
def task(sleep_time, message):
    # block for a moment
    sleep(sleep_time)
    # display a message
    print(message)

# entry point
if __name__ == '__main__':
    # create a process
    process = Process(target=task, args=(1.5, 'New message from another process'))
    # run the process
    process.start()
    # wait for the process to finish
    print('Waiting for the process...')
    process.join()
```

--------------------------------

### Get Main Process Instance from Child Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates how to obtain the `multiprocessing.Process` instance of the main process when executing code within a child process. This is accomplished using the `multiprocessing.parent_process()` function.

```python
# SuperFastPython.com
# example of getting the main process from a child of the main process
from multiprocessing import parent_process
from multiprocessing import Process

# function executed in a child process
def task():
    # get the parent process instance
    process = parent_process()
    # report details of the main process
    print(process)

# entry point
if __name__ == '__main__':
    # create a new process
    process = Process(target=task)
    # start the new process
    process.start()
    # wait for the new process to terminate
    process.join()
```

--------------------------------

### List Active Child Processes in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to retrieve a list of all currently running child processes spawned by the parent process using `multiprocessing.active_children()`. The function returns a list of `Process` objects. The example shows how to count and report details of these active processes.

```python
# SuperFastPython.com
# list all active child processes
from multiprocessing import active_children

# get a list of all active child processes
children = active_children()

# report a count of active children
print(f'Active Children Count: {len(children)}')

# report each in turn
for child in children: print(child)
```

```python
# SuperFastPython.com
# list all active child processes
from time import sleep
from multiprocessing import active_children
from multiprocessing import Process

# function to execute in a new process
def task():
    # block for a moment
    sleep(1)

# entry point
if __name__ == '__main__':
    # create a number of child processes
    processes = [Process(target=task) for _ in range(5)]

    # start the child processes
    for process in processes:
        process.start()

    # get a list of all active child processes
    children = active_children()

    # report a count of active children
    print(f'Active Children Count: {len(children)}')

    # report each in turn
    for child in children:
        print(child)
```

--------------------------------

### Python Multiprocessing Mutex Lock Example

Source: https://superfastpython.com/multiprocessing-in-python/index

This Python code demonstrates the use of multiprocessing.Lock to ensure that only one process can access a critical section at a time. It defines a task function that acquires the lock, prints a message, sleeps for a random duration, and then releases the lock. Multiple processes are created, each targeting this function with the shared lock, and then started and joined.

```python
# SuperFastPython.com
# example of a mutual exclusion (mutex) lock for processes
from time import sleep
from random import random
from multiprocessing import Process
from multiprocessing import Lock

# work function
def task(lock, identifier, value):
    # acquire the lock
    with lock:
        print(f'>process {identifier} got the lock, sleeping for {value}')
        sleep(value)

# entry point
if __name__ == '__main__':
    # create the shared lock
    lock = Lock()

    # create a number of processes with different sleep times
    processes = [Process(target=task, args=(lock, i, random())) for i in range(10)]

    # start the processes
    for process in processes:
        process.start()

    # wait for all processes to finish
    for process in processes:
        process.join()

```

--------------------------------

### Create a Multiprocessing Barrier in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to create a basic multiprocessing Barrier instance. The barrier is configured to wait for a specified number of parties (processes) to arrive before releasing them.

```python
import multiprocessing

# Create a barrier that waits for 10 processes
barrier = multiprocessing.Barrier(10)
```

--------------------------------

### Using Context Manager for Locks in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates the recommended and concise way to manage locks using Python's context manager protocol ('with' statement). This ensures the lock is automatically acquired upon entering the block and released upon exiting, regardless of how the block is exited.

```python
import multiprocessing

lock = multiprocessing.Lock()

# acquire the lock with context manager
with lock:
    # critical section
    print("Inside critical section")

# lock is automatically released here
```

--------------------------------

### Query Process PID in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieve the unique process identifier (PID) assigned by the operating system to a multiprocessing.Process. The PID is only available after the process has been started using the start() method.

```python
# SuperFastPython.com
# example of reporting the native process identifier
from multiprocessing import Process

# entry point
if __name__ == '__main__':
    # create the process
    process = Process()
    # report the process identifier
    print(process.pid)
    # start the process
    process.start()
    # report the process identifier
    print(process.pid)
```

--------------------------------

### Get Parent Process of Main Process in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to use `multiprocessing.parent_process()` to get the parent process of the main process. The main process does not have a parent, so this function returns None.

```python
# SuperFastPython.com
# example of getting the parent process of the main process
from multiprocessing import parent_process

# get the the parent process
process = parent_process()

# report details
print(process)
```

--------------------------------

### Create a Multiprocessing Semaphore in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to initialize a `multiprocessing.Semaphore` with a specified limit for concurrent access. This sets the maximum number of processes that can acquire the semaphore simultaneously.

```python
import multiprocessing

# create a semaphore with a limit of 100
semaphore = multiprocessing.Semaphore(100)
```

--------------------------------

### Get Name of Main Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to retrieve and print the name of the main process in Python. This involves getting the `multiprocessing.Process` instance using `multiprocessing.current_process()` and accessing its 'name' attribute.

```python
# SuperFastPython.com
# example of reporting the name of the main process
from multiprocessing import current_process

# get the main process
process = current_process()

# report the name of the main process
print(process.name)
```

--------------------------------

### Manual Lock Acquisition and Release in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates the traditional method of manually acquiring and releasing a lock using acquire() and release() methods. This approach is prone to errors if the release() call is missed, especially during exceptions.

```python
import multiprocessing

lock = multiprocessing.Lock()

# acquire the lock manually
lock.acquire()

# critical section
print("Inside critical section")

# release the lock
lock.release()
```

--------------------------------

### Configure Barrier with Action and Timeout in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to create a multiprocessing Barrier with an optional action to be executed when all parties arrive and a default timeout for the wait operation. The action is a callable executed by one process before others are released.

```python
import multiprocessing

def my_function():
    print("All processes have arrived!")

# Create a barrier with 10 parties, an action, and a 5-second timeout
barrier = multiprocessing.Barrier(10, action=my_function, timeout=5)
```

--------------------------------

### Get Current Process PID using os.getpid()

Source: https://superfastpython.com/multiprocessing-in-python/index

Obtains the Process ID (PID) of the currently executing process using the `os.getpid()` function. This is a direct way to get the PID of the current script or process without needing a `multiprocessing.Process` object.

```python
import os

# Get the PID for the current process
pid = os.getpid()
print(f"Current Process PID: {pid}")
```

--------------------------------

### Communicate Between Processes using multiprocessing.Queue

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates how to use `multiprocessing.Queue` to return values from a process. A queue provides a FIFO (First-In, First-Out) data structure for inter-process communication. Items are added using `put()` and retrieved using `get()`. By default, queues are unbounded and blocking operations are used for `put()` and `get()`.

```python
import multiprocessing

# Create an unbounded queue
queue = multiprocessing.Queue()

# Add an item to the queue
item_to_send = "Hello from process"
queue.put(item_to_send)

# Retrieve an item from the queue
received_item = queue.get()
print(f"Received item: {received_item}")
```

--------------------------------

### Use Multiprocessing Semaphore as a Context Manager in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates the preferred way to use semaphores via the `with` statement (context manager). This automatically handles acquiring the semaphore upon entering the block and releasing it upon exiting, ensuring proper resource management.

```python
# acquire the semaphore
with semaphore:
    # ... critical section code ...
```

--------------------------------

### Create a Size-Limited Multiprocessing Queue in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to create a multiprocessing.Queue with a maximum size limit. If the queue is full, calls to put() will block until space is available.

```python
import multiprocessing

# created a size limited queue
queue = multiprocessing.Queue(maxsize=100)
```

--------------------------------

### Wait for Processes to Complete

Source: https://superfastpython.com/multiprocessing-in-python/index

Waits for all previously started child processes to finish their execution. This ensures that the main program does not exit before all tasks are completed.

```python
# wait for child processes to finish
for process in processes:
    process.join()
```

--------------------------------

### Retrieve Item from Multiprocessing Queue (Blocking) in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates retrieving an item from a multiprocessing.Queue. By default, get() blocks until an item is available in the queue.

```python
import multiprocessing

queue = multiprocessing.Queue()
queue.put("data_to_retrieve")

# get an item from the queue
item = queue.get()
```

--------------------------------

### Create a Pipe Connection in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to create a pipe connection between two processes using multiprocessing.Pipe(). This returns two Connection objects that can be used for bidirectional communication.

```python
import multiprocessing

conn1, conn2 = multiprocessing.Pipe()
```

--------------------------------

### Get Parent Process PID using os.getppid()

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves the Process ID (PID) of the parent process using the `os.getppid()` function. This is useful for understanding process relationships and hierarchies.

```python
import os

# Get the PID for the parent process
parent_pid = os.getppid()
print(f"Parent Process PID: {parent_pid}")
```

--------------------------------

### Get Current Process Instance in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves the multiprocessing.Process instance for the currently executing code. This is useful for identifying the main process or a child process. It requires the 'multiprocessing' module.

```python
# SuperFastPython.com
# example of executing the current process
from multiprocessing import current_process

# get the current process
process = current_process()

# report details
print(process)
```

--------------------------------

### Retrieve an Item from a Queue in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to retrieve an item from a multiprocessing Queue using the get() method. This operation follows FIFO order and is process-safe, ensuring data is correctly dequeued.

```python
item = queue.get()
```

--------------------------------

### Acquiring Lock with Timeout in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to use a timeout when acquiring a lock to prevent indefinite blocking. If the lock cannot be acquired within the specified timeout, the acquire() method returns False, allowing for error handling or alternative actions.

```python
import multiprocessing

lock = multiprocessing.Lock()

# acquire the lock with a timeout of 2 minutes
if not lock.acquire(timeout=2*60):
    # handle failure case, e.g., report error or force termination
    print("Failed to acquire lock within timeout")
else:
    # critical section
    print("Lock acquired successfully")
    # ... do work ...
    lock.release()
```

--------------------------------

### Get Parent Process of Child Process in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates how to obtain the parent process of a newly created child process. The child process's parent is the main process, which is correctly identified by `multiprocessing.parent_process()`.

```python
# function to execute in a new process
def task():
    # get the the parent process
    process = parent_process()
    # report details
    print(process)

# entry point
if __name__ == '__main__':
    # create a new process
    process = Process(target=task)
    # start the new process
    process.start()
    # wait for the new process to terminate
    process.join()
```

--------------------------------

### Configure Logger in Each Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

This snippet shows how to configure a logger directly within each process. It sets the logger's level to DEBUG to capture all messages. However, this approach has limitations, including duplicated configuration code and potential data loss or corruption due to concurrent writes.

```python
# create a logger
logger = logging.getLogger()

# log all messages, debug and up
logger.setLevel(logging.DEBUG)
```

--------------------------------

### Create an Unbounded Multiprocessing Queue in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Demonstrates how to create an unbounded queue using multiprocessing.Queue. This queue has no size limit, allowing an unlimited number of items to be added.

```python
import multiprocessing

# created an unbounded queue
queue = multiprocessing.Queue()
```

--------------------------------

### Get Process PID using multiprocessing.Process

Source: https://superfastpython.com/multiprocessing-in-python/index

Retrieves the Process ID (PID) from a `multiprocessing.Process` instance. The PID is a unique identifier assigned by the operating system to a running process. This method requires an active process instance.

```python
import multiprocessing

# Get the current process instance
process = multiprocessing.current_process()

# Get the PID from the process instance
pid = process.pid
print(f"Process PID: {pid}")
```

--------------------------------

### Define Custom Process Class by Extending multiprocessing.Process (Python)

Source: https://superfastpython.com/multiprocessing-in-python/index

Illustrates the basic structure for creating a custom process class by inheriting from multiprocessing.Process. This involves defining a new class that extends the base Process class and overriding the run() method to contain the code that will be executed in the new process. Constructors and additional methods can be added for more complex scenarios.

```python
# custom process class
class CustomProcess(multiprocessing.Process):
    # ...

    # override the run function
    def run(self):
        # ...
        pass
```

--------------------------------

### Acquire a Multiprocessing Semaphore in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Shows how to acquire a semaphore using the `acquire()` method. By default, this is a blocking call, meaning the process will wait until the semaphore is available. It returns `True` if acquired, `False` otherwise.

```python
# acquire the semaphore
semaphore.acquire()
```

--------------------------------

### Query Process Alive Status in Python

Source: https://superfastpython.com/multiprocessing-in-python/index

Check if a multiprocessing.Process is currently executing its run() method. The is_alive() method returns True if the process is running and False otherwise. It's False before start() is called and after run() completes.

```python
# SuperFastPython.com
# example of assessing whether a process is alive
from multiprocessing import Process

# entry point
if __name__ == '__main__':
    # create the process
    process = Process()
    # report the process is alive
    print(process.is_alive())
```
Tokens:
Raw