ipyparallel

https://ipyparallel.readthedocs.io/en/latest/
Admin
A standalone Python package that enables parallel computing by distributing tasks across a cluster...
Tokens:186,690
Snippets:1,038
Trust Score:8.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Start and Connect to IPython Parallel Cluster in One Line

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Provides a concise way to start an IPython parallel cluster with a specified number of engines and immediately connect a client to it using a single synchronous method call, simplifying the setup process.

```Python
import ipyparallel as ipp
rc = ipp.Cluster(n=4).start_and_connect_sync()
```

--------------------------------

### Starting and Connecting to IPyParallel Cluster Synchronously

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Provides a concise one-liner to both start an IPyParallel cluster with a specified number of engines and immediately connect a client to it synchronously, returning the connected client object.

```ipython
In [1]: import ipyparallel as ipp
In [2]: rc = ipp.Cluster(n=4).start_and_connect_sync()
```

--------------------------------

### PBS Engine Batch Script Template Command Example

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

An example command line for the PBS engine batch script template, demonstrating the use of placeholders like '{n}' for the number of engines. This command is executed by mpiexec to start ipengine instances.

```Bash
/usr/local/bin/mpiexec -n {n} ipengine --debug
```

--------------------------------

### Programmatically Starting an IPyParallel Cluster

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Demonstrates how to initialize and start an IPyParallel cluster with a specified number of engines using the `ipyparallel` library. It shows both asynchronous and synchronous methods for cluster startup, emphasizing the `_sync` suffix for blocking versions of async methods.

```ipython
In [1]: import ipyparallel as ipp
In [2]: cluster = ipp.Cluster(n=4)
In [3]: await cluster.start_cluster() # or cluster.start_cluster_sync() without await
```

--------------------------------

### Install ipyparallel Package

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Instructions for installing the ipyparallel Python package using either pip or conda, which are common package managers for Python environments.

```Bash
pip install ipyparallel
```

```Bash
conda install ipyparallel
```

--------------------------------

### Start IPython Parallel Cluster Programmatically

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Demonstrates how to initialize and start an IPython parallel cluster with a specified number of engines using Python code. It shows both asynchronous (`await cluster.start_cluster()`) and synchronous (`cluster.start_cluster_sync()`) methods for starting the cluster.

```Python
import ipyparallel as ipp
cluster = ipp.Cluster(n=4)
await cluster.start_cluster() # or cluster.start_cluster_sync() without await
```

--------------------------------

### Create IPython Profile for Parallel Computing

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Command-line instruction to create a new IPython profile specifically configured for parallel computing, often used as a starting point for custom cluster setups.

```Bash
$ ipython profile create --parallel --profile=ssh
```

--------------------------------

### Starting IPyParallel Cluster via Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Illustrates how to launch an IPyParallel cluster directly from the command line using the `ipcluster` utility, specifying the number of engines to start. This method is equivalent to programmatically calling `ipp.Cluster(n=4, cluster_id="").start_cluster()`.

```shell
$ ipcluster start -n 4
```

--------------------------------

### IPyParallel Core Classes and Methods

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Documentation for key classes and methods within the `ipyparallel` library, including `Client` and `Cluster`. It describes their purpose, how they are used for connecting to and managing parallel computing resources, and their behavior as context managers.

```APIDOC
ipyparallel.Client:
  - Primary object for connecting to an IPyParallel cluster.
  - Can be used as a context manager: exiting closes the client's socket connections.
  - `connect_client_sync()`: Synchronously connects a client to a cluster.
  - `from_file(path: str = None, profile: str = None, cluster_id: str = None, sshserver: str = None)`:
    - Connects to an already-running cluster by loading connection details from a JSON file.
    - Parameters:
      - `path`: Optional. Path to the cluster JSON file. If not provided, tries to find it in `~/.ipython/profile_default/security`.
      - `profile`: Optional. The IPython profile to use.
      - `cluster_id`: Optional. The ID of the cluster to connect to.
      - `sshserver`: Optional. SSH server address (e.g., 'me@myhub.example.com') for tunneling access to a remote Hub.

ipyparallel.Cluster:
  - Object for managing and interacting with an IPyParallel cluster.
  - Can be used as a context manager: entering starts the cluster, waits for engines, and connects a client (returning the client); exiting shuts down all cluster resources.
  - `__init__(n: int = 0, cluster_id: str = "")`:
    - Initializes a Cluster object.
    - Parameters:
      - `n`: Number of engines to start.
      - `cluster_id`: Identifier for the cluster.
  - `start_cluster()`: Asynchronously starts the cluster.
  - `start_cluster_sync()`: Synchronously starts the cluster (blocking version of `start_cluster()`).
  - `start_and_connect_sync(n: int = 0)`:
    - Synchronously starts a cluster and connects a client in one step.
    - Parameters:
      - `n`: Number of engines to start.
    - Returns: The connected `Client` object.
  - `from_file(path: str = None, profile: str = None, cluster_id: str = None)`:
    - Connects to an already-running cluster by loading connection details from a JSON file.
    - Parameters: Same as `Client.from_file`.

ipyparallel.DirectView:
  - A view class for explicit addressing of engines within the cluster.

ipyparallel.LoadBalancedView:
  - A view class for destination-agnostic scheduling of tasks across engines.
```

--------------------------------

### Basic IPyParallel Client Connection and Interaction

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Demonstrates the fundamental steps of connecting a client to an existing cluster and performing a basic distributed operation. It shows how to wait for engines, retrieve engine IDs, and apply a simple function across all engines.

```ipython
In [2]: rc = cluster.connect_client_sync()

In [3]: rc.wait_for_engines(n=4)

In [4]: rc.ids
Out[4]: [0, 1, 2, 3]

In [5]: rc[:].apply_sync(lambda: "Hello, World")
Out[5]: [ 'Hello, World', 'Hello, World', 'Hello, World', 'Hello, World' ]
```

--------------------------------

### Starting a Local IPython Parallel Cluster

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

Demonstrates how to programmatically initialize and start an IPython Parallel cluster locally with a specified number of engines. This setup is ideal for testing or running on a multicore machine without complex configurations.

```python
cluster = ipp.Cluster(n=4)
cluster.start_cluster_sync()
```

--------------------------------

### Start ipyparallel Controller Listening on Specific IP

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Shows how to start the ipyparallel controller, making it listen on a specific IP address visible to engine machines. This is a crucial first step for engines to connect to the controller in a distributed setup, ensuring the controller is accessible from remote hosts.

```bash
[controller.host] $ ipcontroller --ip=192.168.1.16
```

--------------------------------

### IPYparallel Setup and Basic Usage

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/examples/Monitoring an MPI Simulation - 2.ipynb

Demonstrates how to set up an IPYparallel cluster and execute simple tasks in parallel. This includes starting the controller and engines, and submitting a basic function call.

```python
from ipyparallel import Client

# Start a controller (if not already running)
# $ ipcluster start

# Connect to the controller
c = Client()

# View connected engines
print(c.ids)

# Submit a simple function to all engines
results = c[:].apply(lambda x: x + 1, 5)

# Print results
print(results.get())
```

```bash
# Start the IPYparallel cluster
ipcluster start
```

--------------------------------

### Start ipyparallel Engine with Explicit Connection File

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Demonstrates how to start an ipyparallel engine by explicitly providing the path to the connection file using the `--file` argument. This file, typically obtained from the controller, contains all the necessary information for the engine to connect and register itself.

```bash
[engine.host.n] $ ipengine --file=./ipcontroller-engine.json
```

--------------------------------

### Start IPyparallel Engine with Connection File

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

Illustrates how to launch an IPyparallel engine by explicitly providing the path to the `ipcontroller-engine.json` connection file, which contains the necessary information to connect to the controller.

```bash
$ ipengine --file=/path/to/my/ipcontroller-engine.json
```

--------------------------------

### Start ipcluster with MPI Engines (Command Line)

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Command-line instruction to start an IPython cluster, specifically configuring it to use MPI for engine startup. This method is suitable when the default MPI configuration is sufficient and MPI is installed and configured on the system.

```Bash
$> ipcluster start --engines=mpi
```

--------------------------------

### Start IPython Parallel Cluster from Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Illustrates how to launch an IPython parallel cluster with a specified number of engines directly from the command line using the `ipcluster` utility. This method is equivalent to programmatically starting a cluster with a default cluster ID.

```Shell
ipcluster start -n 4
```

--------------------------------

### Start Local IPython Cluster Programmatically

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

This Python snippet demonstrates how to use the `ipp.Cluster` class to programmatically start a local IPython controller and a specified number of engines. This is particularly useful for testing, development, or running parallel computations on a multicore machine without manual command-line setup.

```python
cluster = ipp.Cluster(n=4)
cluster.start_cluster_sync()
```

--------------------------------

### Start ipyparallel Engine Using a Specific Profile with Copied Connection File

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Shows an alternative method for connecting engines by copying the connection file directly into a specific ipyparallel profile's security directory. This allows starting the engine simply by specifying the profile name, without needing to explicitly point to the connection file.

```bash
[engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ~/.ipython/profile_ssh/security/
[engine.host.n] $ ipengine --profile=ssh
```

--------------------------------

### Configure IPython Engine Startup Scripts and Commands

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

These configurations allow specifying a Python script file or a direct Python command to be executed at the startup of the IPython Engine. This is useful for importing necessary libraries, setting up environment variables, or running any initial setup code required for the engine's operation.

```Python
c.IPEngine.startup_script = u'/path/to/my/startup.py'

c.IPEngine.startup_command = 'import numpy, scipy, mpi4py'
```

--------------------------------

### ipyparallel Cluster.start_and_connect Method

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/changelog.md

The `Cluster.start_and_connect` method provides a convenient way to start an ipyparallel cluster and connect to it in a single operation. The `activate` parameter allows for immediate activation of `%px` magics, streamlining the setup process for parallel execution.

```APIDOC
Cluster.start_and_connect(activate=True)
  - Parameters:
    - activate (bool): If True, automatically activates the `%px` magics for immediate use after connection. Defaults to False if not specified.
  - Returns: A connected Cluster object.
  - Usage: Simplifies the startup and configuration of an ipyparallel cluster for interactive sessions.
```

--------------------------------

### Make IPyParallel Controller JSON Persistent

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This command-line example shows how to start the `ipcontroller` with the `--reuse` flag. This flag ensures that the connection information in the generated JSON files remains accurate across controller restarts, eliminating the need to regenerate them each time.

```bash
$ ipcontroller --reuse
```

--------------------------------

### Example ipcontroller-client.json with SSH tunnel and location

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

This JSON snippet shows an example `ipcontroller-client.json` file. It reflects the configuration of an `ipcontroller` set up with an SSH server for tunneling (`login.mycluster.net`) and a specified external location (`10.0.1.5`). This file contains all necessary information for a client to connect to the controller, including the URL, execution key, SSH server, and location.

```json
{
  "url":"tcp:\/\/*:43447",
  "exec_key":"9c7779e4-d08a-4c3b-ba8e-db1f80b562c1",
  "ssh":"login.mycluster.net",
  "location":"10.0.1.5"
}
```

--------------------------------

### iPyParallel Launcher API: Start and Stop Methods

Source: https://ipyparallel.readthedocs.io/en/latest/reference/launchers

This API documentation outlines the expected signatures and behaviors for the `start` and `stop` methods across iPyParallel launcher implementations. The `start` method initiates processes, while `stop` handles their graceful termination and resource cleanup. Specific launcher types may have variations in parameters, such as `EngineLauncher.start` accepting an engine count.

```APIDOC
BaseLauncher.start()
  - Description: Requests the process(es) to be started and initiates monitoring for process exit. Must call `notify_stop()` upon exit.
  - Parameters:
    - self.args (list): The command arguments to launch the process.
  - Returns: None

ControllerLauncher.start()
  - Description: Starts the iPyParallel controller process.
  - Parameters: None
  - Returns: None

EngineLauncher.start(n: Optional[int] = None)
  - Description: Starts engine processes. If `n` is an integer, starts that many engines. If `n` is `None`, a default number (e.g., number of CPUs) should be used.
  - Parameters:
    - n (int | None): The number of engines to start.
  - Returns: None

BaseLauncher.stop()
  - Description: Requests that the process(es) stop and returns only after all processes are stopped and resources are cleaned up. The cleanup mechanism depends on how resources were requested in `start`.
  - Parameters: None
  - Returns: None

```

--------------------------------

### Start ipyparallel Cluster with Controller and Engines (Convenience Method)

Source: https://ipyparallel.readthedocs.io/en/latest/examples/Cluster%20API

Presents a convenience method, `start_cluster()`, which asynchronously starts both the controller and a specified number of engines in a single call. This simplifies the initial setup of a cluster.

```Python
engine_set_id = await cluster.start_cluster(n=4)
```

--------------------------------

### Using IPyParallel Cluster as a Context Manager

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Illustrates how to use the `ipyparallel.Cluster` class as a context manager for automatic resource management. When entering the context, it starts the cluster, waits for engines, and connects a client. Upon exiting, it ensures the cluster resources are properly shut down, simplifying cleanup.

```python
import ipyparallel as ipp
# start cluster, connect client
with ipp.Cluster(n=4) as rc:
    e_all = rc[:]
    ar = e_all.apply_sync(task)
    ar.wait_interactive()
    results = ar.get()
# have results, cluster is shutdown
```

--------------------------------

### Manage IPython Parallel Cluster and Client with Context Managers

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Illustrates the use of Python context managers (`with` statements) for `ipp.Cluster` and `ipp.Client` classes. This ensures automatic startup, client connection, and proper shutdown of cluster resources upon exiting the context, simplifying resource management and cleanup.

```Python
import ipyparallel as ipp
# start cluster, connect client
with ipp.Cluster(n=4) as rc:
    e_all = rc[:]
    ar = e_all.apply_sync(task)
    ar.wait_interactive()
    results = ar.get()
# have results, cluster is shutdown
```

--------------------------------

### Start IPython Cluster from Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

A command-line instruction to start an IPython cluster using a specified profile and number of engines. This is an alternative to programmatic cluster management.

```Bash
$ ipcluster start --profile=pbs -n 128
```

--------------------------------

### Python LocalProcessLauncher Start Method Implementation

Source: https://ipyparallel.readthedocs.io/en/latest/reference/launchers

This Python code defines the `start` method for a `LocalProcessLauncher`. It handles the initiation of a local process, sets up its environment, redirects standard output and error, and begins monitoring the process using `psutil`. It ensures the launcher is in the correct state before attempting to start the process.

```python
    def start(self):
        self.log.debug("Starting %s: %r", self.__class__.__name__, self.args)
        if self.state != 'before':
            raise ProcessStateError(
                'The process was already started and has state: {self.state}'
            )
        self.log.debug(f"Sending output for {self.identifier} to {self.output_file}")

        env = os.environ.copy()
        env.update(self.get_env())
        self.log.debug(f"Setting environment: {','.join(self.get_env())}")

        with open(self.output_file, "ab") as f, open(os.devnull, "rb") as stdin:
            proc = self._popen_process = Popen(
                self.args,
                stdout=f.fileno(),
                stderr=STDOUT,
                stdin=stdin,
                env=env,
                cwd=self.work_dir,
                start_new_session=True,  # don't forward signals
            )
        self.pid = proc.pid
        # use psutil API for self.process
        self.process = psutil.Process(proc.pid)

        self.notify_start(self.process.pid)
        self._start_waiting()
        if 1 <= self.log.getEffectiveLevel() <= logging.DEBUG:
            self._start_streaming()

```

--------------------------------

### Connecting to IPyParallel Cluster

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/intro.md

Provides various methods for connecting a client to an existing IPyParallel cluster. This includes connecting from a default or specified configuration file, connecting with a profile, and connecting via an SSH server for remote clusters.

```python
cluster = ipp.Cluster.from_file()
```

```ipython
In [2]: cluster = ipp.Cluster.from_file(profile="myprofile", cluster_id="...")
In [3]: rc = cluster.connect_client_sync()
```

```ipython
In [2]: cluster = ipp.Cluster.from_file('/path/to/my/cluster-.json')
```

```ipython
In [2]: c = ipp.Client('/path/to/my/ipcontroller-client.json', sshserver='me@myhub.example.com')
```

--------------------------------

### Start IPython Engine with Copied Connection File and Profile

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This sequence of commands demonstrates an alternative method for connecting an engine. It first copies the controller's connection file into a specific IPython profile's security directory on the engine host, then starts the engine using that profile. This avoids needing to specify the file path directly on subsequent runs if the profile is reused.

```bash
[engine.host.n] $ scp controller.host:.ipython/profile_default/security/ipcontroller-engine.json ~/.ipython/profile_ssh/security/
[engine.host.n] $ ipengine --profile=ssh
```

--------------------------------

### Start ipcontroller with Specific IP Binding

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Demonstrates how to start the ipcontroller process, instructing it to listen on a specific IP address visible to engine machines. This can be done either via a command-line argument or by setting the 'IPController.ip' configuration in 'ipcontroller_config.py'.

```shell
$ ipcontroller --ip=192.168.1.16
```

```python
# in ipcontroller_config.py
IPController.ip = '192.168.1.16'
```

--------------------------------

### Start IPyparallel Controller with Specific IP

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

Demonstrates how to start the IPyparallel controller, explicitly binding it to a specific IP address visible to engine machines. This can be done via a command-line argument or within the `ipcontroller_config.py` file.

```bash
$ ipcontroller --ip=192.168.1.16
```

```python
# in ipcontroller_config.py
IPController.ip = '192.168.1.16'
```

--------------------------------

### Start ipengine with Explicit Connection File Path

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Illustrates how to launch an ipengine process, explicitly specifying the path to the 'ipcontroller-engine.json' connection file. This is necessary when the file is not located in the default 'IPYTHONDIR/profile_<name>/security' directory on the engine's host.

```shell
$ ipengine --file=/path/to/my/ipcontroller-engine.json
```

--------------------------------

### Configure IPython Controller with Scheduling Schemes

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/task

This snippet demonstrates how to start the IPython controller and specify a task scheduling scheme using the `--scheme` command-line argument. It provides an example of selecting the 'lru' (Least Recently Used) scheme.

```Shell
$ ipcontroller --scheme=<schemename>
for instance:
$ ipcontroller --scheme=lru
```

--------------------------------

### Connect Client and Verify IPython Parallel Cluster Functionality

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Demonstrates how to connect a client to an existing IPython parallel cluster, wait for the specified number of engines to become ready, retrieve their IDs, and execute a simple parallel task across all engines to verify the cluster's functionality.

```Python
rc = cluster.connect_client_sync()

rc.wait_for_engines(n=4)

rc.ids

rc[:].apply_sync(lambda: "Hello, World")
```

--------------------------------

### Start IPython Cluster from Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This bash command initiates an IPython cluster from the command line, specifying the profile to use (e.g., 'pbs') and the desired number of engines to launch. It's a common way to manually start a cluster.

```bash
ipcluster start --profile=pbs -n 128
```

--------------------------------

### Start IPython Parallel Cluster Components

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/examples/Cluster API.ipynb

These snippets illustrate how to start various components of an IPython Parallel cluster. This includes asynchronously starting the controller, asynchronously starting a set of engines, and synchronously starting another set of engines. A convenience method `start_cluster` is also shown for starting both the controller and engines in one call.

```Python
await cluster.start_controller()
cluster
```

```Python
engine_set_id = await cluster.start_engines(n=4)
cluster
```

```Python
engine_set_2 = cluster.start_engines_sync(n=2)
engine_set_2
```

```Python
engine_set_id = await cluster.start_cluster(n=4)
```

--------------------------------

### Start ipyparallel Engine with SSH Tunnel via Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Demonstrates how to start an ipyparallel engine and establish an SSH tunnel to a specified login node using the `--ssh` argument. This is useful when the engine is not on the same LAN as the controller or in restricted network environments, allowing the engine to connect through an SSH gateway.

```bash
ipengine --profile=foo --ssh=my.login.node
```

--------------------------------

### Start IPython Engine with Explicit Connection File

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This command starts an IPython engine instance, explicitly pointing it to a connection file. The engine reads this file to obtain the necessary information to connect and register with the IPython controller, enabling distributed computation.

```bash
[engine.host.n] $ ipengine --file=./ipcontroller-engine.json
```

--------------------------------

### Configure IPython Parallel for Restricted Network with SSH

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This example illustrates a setup for IPython Parallel in a highly restricted network where all connections are loopback. It shows how to start the ipcontroller with an SSH server specified, followed by starting individual ipengine instances and using ipcluster to manage multiple engines, all leveraging SSH tunnels for communication.

```bash
[node1] $> ipcontroller --enginessh=node1
[node2] $> ipengine
[node3] $> ipcluster engines --n=4
```

--------------------------------

### Retrieve Parallel Results as Dictionary with ipyparallel AsyncResult

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/asyncresult

This Python example demonstrates how to use the `AsyncResult.get_dict()` method to retrieve results from parallel computations, keyed by engine ID. It shows a practical application where each engine gets a dictionary mapping PIDs of all other engines, useful for inter-engine communication setup.

```Python
In [10]: ar = rc[:].apply_async(os.getpid)
In [11]: pids = ar.get_dict()
In [12]: rc[:]['pid_map'] = pids
```

--------------------------------

### Initialize IPython Parallel Cluster with MPI (Python)

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Python code to initialize an IPython Parallel cluster, specifying 'mpi' as the engine type and requesting 4 engines. It then starts the cluster and connects synchronously, which involves launching the IPython controller and using 'mpiexec' to start the specified number of engines.

```Python
cluster = ipp.Cluster(engines="mpi", n=4)
client = cluster.start_and_connect_sync()
```

--------------------------------

### Start and Connect ipyparallel Cluster with Magics

Source: https://ipyparallel.readthedocs.io/en/latest/api/ipyparallel

Demonstrates how to asynchronously start an ipyparallel cluster with a specified number of engines and connect to it, automatically activating `%px` magics for direct execution on all engines.

```python
rc = await Cluster(engines="mpi").start_and_connect(n=8, activate=True)

%px print("hello, world!")
```

--------------------------------

### Make ipcontroller JSON files persistent

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

This command demonstrates how to start the `ipcontroller` with the `--reuse` flag. This flag ensures that the connection information in the JSON files remains accurate across sessions, allowing you to create the key (JSON file) once and reuse it for future connections without regeneration.

```bash
ipcontroller --reuse
```

--------------------------------

### Start IPython Cluster with MPI Launcher

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/reference/mpi.md

This snippet demonstrates two ways to start an IPython cluster with MPI-enabled engines. The Python code uses `ipp.Cluster` to configure and start the cluster, while the command-line example shows how to achieve the same using `ipcluster start`. Both methods ensure that engines are launched via `mpiexec` and `MPI_Init` is called.

```python
cluster = ipp.Cluster(engines="mpi")
cluster.start_cluster_sync()
```

```bash
ipcluster start -n 4 --engines=mpi
```

--------------------------------

### ipyparallel Cluster Launcher: `start()` Method

Source: https://ipyparallel.readthedocs.io/en/latest/genindex

Documents the `start()` method, a crucial function implemented across various `ipyparallel.cluster.launcher` classes. This method initiates the launching process for controllers, engines, or entire clusters, depending on the specific launcher type.

```APIDOC
ipyparallel.cluster.launcher.BaseLauncher.start()
ipyparallel.cluster.launcher.BatchControllerLauncher.start()
ipyparallel.cluster.launcher.BatchSystemLauncher.start()
ipyparallel.cluster.launcher.LocalControllerLauncher.start()
ipyparallel.cluster.launcher.LocalEngineSetLauncher.start()
ipyparallel.cluster.launcher.LocalProcessLauncher.start()
ipyparallel.cluster.launcher.LSFLauncher.start()
ipyparallel.cluster.launcher.MPIEngineSetLauncher.start()
ipyparallel.cluster.launcher.MPILauncher.start()
ipyparallel.cluster.launcher.SSHEngineSetLauncher.start()
ipyparallel.cluster.launcher.SSHLauncher.start()
ipyparallel.cluster.launcher.SSHProxyEngineSetLauncher.start()
ipyparallel.cluster.launcher.WindowsHPCEngineSetLauncher.start()
ipyparallel.cluster.launcher.WindowsHPCLauncher.start()
```

--------------------------------

### Start IPython Cluster with MPI Engines

Source: https://ipyparallel.readthedocs.io/en/latest/reference/mpi

These snippets demonstrate how to start an IPython cluster configured to use MPI for its engines. The first example uses the Python API to programmatically initialize and start the cluster, while the second shows the equivalent command-line invocation using `ipcluster`.

```python
cluster = ipp.Cluster(engines="mpi")
cluster.start_cluster_sync()
```

```bash
ipcluster start -n 4 --engines=mpi
```

--------------------------------

### Configure IPyParallel Engine Startup Scripts and Working Directory

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

These configurations allow specifying a Python file or a command string to be executed when the IP Engine starts up, useful for importing common libraries or setting up the environment. Additionally, `c.IPEngine.work_dir` can set a scratch directory for the engine, particularly useful on systems with shared filesystems.

```python
c.IPEngine.startup_script = u'/path/to/my/startup.py'
c.IPEngine.startup_command = 'import numpy, scipy, mpi4py'
c.IPEngine.work_dir = u'/path/to/scratch/'
```

--------------------------------

### Query ipyparallel Jobs Started by Current User in Last Hour

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/reference/db.md

This example demonstrates how to query the ipyparallel database for jobs started by the current client within the last hour. It involves calculating a time threshold using `datetime` and `timedelta`, then filtering by `started` time using the `$gte` operator and matching the `client_uuid` to the current session.

```ipython
In [1]: from datetime import datetime, timedelta

In [2]: hourago = datetime.now() - timedelta(1./24)

In [3]: recent = rc.db_query({'started' : {'$gte' : hourago },
                                'client_uuid' : rc.session.session})
```

--------------------------------

### Start IPython with Matplotlib Integration

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/demos.md

This shell command launches the IPython interactive shell with Matplotlib integration enabled, allowing for immediate plotting capabilities within the session. It's typically used when starting a new IPython session from the command line.

```bash
ipython --matplotlib
```

--------------------------------

### ipyparallel.cluster.launcher.SSHEngineSetLauncher API

Source: https://ipyparallel.readthedocs.io/en/latest/reference/launchers

API documentation for the `SSHEngineSetLauncher` class, detailing methods for starting engines and configuration properties related to process termination timeouts and file transfer lists.

```APIDOC
SSHEngineSetLauncher:
  start(*n*)
    Description: Start engines by profile or profile_dir.
    Parameters:
      n: An upper limit of engines. The `engines` config property is used to assign slots to hosts.

  stop_seconds_until_kill: Int(5)
    Description: The number of seconds to wait for a process to exit after sending SIGTERM before sending SIGKILL.

  stop_timeout: Int(60)
    Description: The number of seconds to wait for a process to exit before raising a TimeoutError in stop.

  to_fetch: List()
    Description: List of (remote, local) files to fetch after starting.

  to_send: List()
    Description: List of (local, remote) files to send before starting.

  user: Unicode('')
    Description: Username for ssh.
```

--------------------------------

### Configure ipyparallel for SSH Tunnels with `ipcontroller` and `ipcluster`

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Illustrates a scenario where all connections are loopback and SSH tunnels are used to connect engines to the controller. It shows starting an `ipcontroller` with an SSH target, then an `ipengine`, and finally scaling engines using `ipcluster` for a fully restricted system setup.

```bash
[node1] $> ipcontroller --enginessh=node1
[node2] $> ipengine
[node3] $> ipcluster engines --n=4
```

--------------------------------

### Initialize IPython Parallel Cluster and Client

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/task

This snippet demonstrates the initial steps to set up an IPython parallel computing environment. It involves importing the `ipyparallel` library, starting a cluster, connecting a client, and waiting for a specified number of engines to become available.

```python
import ipyparallel as ipp
cluster = ipp.Cluster()
cluster.start_cluster_sync()
rc = cluster.connect_client_sync()
rc.wait_for_engines(4)
```

--------------------------------

### Command to Start ipcontroller for Remote Cluster Access

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/reference/security.md

This command-line instruction demonstrates how to start the `ipcontroller` process configured to accept connections from any IP address (`--ip=*`) and to use a specified SSH gateway (`--ssh=login.mycluster.com`). This setup is typical for running the controller on a work node within a cluster, allowing clients to connect via SSH tunnels from external machines.

```bash
$> ipcontroller --ip=* --ssh=login.mycluster.com
```

--------------------------------

### Execute Python Script in IPython

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/demos.md

Executes a Python script named `pidigits.py` within the IPython environment. This script likely contains function definitions or setup code required for the subsequent parallel computations.

```python
run pidigits.py
```

--------------------------------

### Starting IPython Parallel Cluster with MPI Engines via Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

Demonstrates how to quickly launch an IPython Parallel cluster with engines managed by MPI directly from the command line. This method assumes a sufficient default MPI configuration is in place.

```bash
ipcluster start --engines=mpi
```

--------------------------------

### Create IPython Parallel Profile for Batch Systems (Command Line)

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

Command-line instruction to create a new IPython profile specifically for parallel computing with batch systems like PBS. This profile will be used to store related configuration settings, allowing for more extensive and organized configurations.

```Bash
$ ipython profile create --parallel --profile=pbs
```

--------------------------------

### Executing Functions on Engines via DirectView

Source: https://ipyparallel.readthedocs.io/en/latest/reference/details

Illustrates how to execute functions on remote engines using the DirectView's `apply_sync` method. Examples include getting the process ID and hostname of the machines where the engines are running.

```python
import os

dview.apply_sync(os.getpid)
```

```python
import socket

dview.apply_sync(socket.gethostname)
```

--------------------------------

### Sample PBS Batch Script Template for IPython Engines

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This comprehensive bash script serves as a template for submitting IPython engine jobs to a PBS cluster. It defines job parameters like name, walltime, node allocation, and includes the command to start `ipengine`, demonstrating variable substitution for dynamic values like the number of engines (`{n}`) and queue (`{queue}`).

```bash
#!/bin/bash
#PBS -N ipython-parallel  # set the name of the job (convenient)
#PBS -V  # export environment variables, required
#PBS -j oe  # merge stdout and error
#PBS -o {output_file}  # send output to a file
#PBS -l walltime=00:10:00  # max runtime 10 minutes
#PBS -l nodes={n//4}:ppn=4  # 4 processes per node
#PBS -q {queue}  # run on the specified queue

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
/usr/local/bin/mpiexec -n {n} {program_and_args}  # start the configured program
# program_and_args is populated via `engine_args` and other configuration
# or you can ignore that and configure the full program in the template
/usr/local/bin/mpiexec -n {n} ipengine --debug
```

--------------------------------

### Start IPython with Matplotlib Integration

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/demos

This command is executed at the system command line to launch the IPython interactive shell. The `--matplotlib` flag automatically enables Matplotlib integration, allowing users to create and display plots directly within the IPython environment.

```bash
ipython --matplotlib
```

--------------------------------

### Configure ipcontroller to listen on all interfaces

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

This Python snippet, intended for `ipcontroller_config.py`, shows how to set the `IPController.ip` trait to `'*'` to instruct the controller to listen on all available network interfaces. By default, the controller listens only on loopback, which is more secure but often impractical for distributed setups.

```python
c.IPController.ip = '*'
```

--------------------------------

### Registering ipyparallel Launchers via Entry Points in setup.py

Source: https://ipyparallel.readthedocs.io/en/latest/reference/launchers

This Python `setup.py` configuration demonstrates how to register custom `ipyparallel` controller and engine launchers using setuptools entry points. This allows `ipyparallel` to discover and use your custom launchers by a short alias (e.g., 'mine') instead of requiring the full import path.

```Python
setup(
    ...
    entry_points={
        'ipyparallel.controller_launchers': [
            'mine = mypackage:MyControllerLauncher',
        ],
        'ipyparallel.engine_launchers': [
            'mine = mypackage:MyEngineSetLauncher',
        ],
    },
)
```

--------------------------------

### Configure ipcontroller database backend

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

This Python snippet demonstrates how to set the `IPController.db_class` trait within `ipcontroller_config.py`. This trait is used to specify the database backend for the Hub, which stores messages and results. The example shows how to configure it for a simple dict-based in-memory implementation.

```python
# for a simple dict-based in-memory implementation, use dictdb
```

--------------------------------

### Implementing Custom IPyParallel Launchers

Source: https://ipyparallel.readthedocs.io/en/latest/reference/launchers

This documentation outlines the essential methods that must be implemented when developing a custom Launcher for IPyParallel. These methods define how the custom launcher will start and stop processes, and how it can be instantiated from a dictionary configuration.

```APIDOC
Methods to Implement for Custom Launchers:

start()
  - Description: The method responsible for initiating the process(es) managed by the custom launcher.
  - Type: async def coroutine.

stop()
  - Description: The method responsible for terminating the process(es) managed by the custom launcher.
  - Type: async def coroutine.

from_dict()
  - Description: A class method or static method that allows the launcher to be instantiated from a dictionary, typically used for configuration loading.
```

--------------------------------

### Sample PBS Batch Script Template for IPython Parallel (Bash)

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

A template for a PBS batch script designed to run IPython Parallel. It includes directives for job naming, environment variable export, output redirection, walltime limits, node and process allocation, and queue selection. The script executes 'mpiexec' to start the configured program within the PBS environment.

```Bash
#!/bin/bash
#PBS -N ipython-parallel  # set the name of the job (convenient)
#PBS -V  # export environment variables, required
#PBS -j oe  # merge stdout and error
#PBS -o {output_file}  # send output to a file
#PBS -l walltime=00:10:00  # max runtime 10 minutes
#PBS -l nodes={n//4}:ppn=4  # 4 processes per node
#PBS -q {queue}  # run on the specified queue

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
/usr/local/bin/mpiexec -n {n} {program_and_args}  # start the configured program
# program_and_args is populated via `engine_args` and other configuration
```

--------------------------------

### Enable IPython Controller Database Backend via Command Line

Source: https://ipyparallel.readthedocs.io/en/latest/reference/db

This command-line example demonstrates how to configure the IPython controller to use a specific database backend (SQLite, MongoDB, or no database) when starting the controller. This determines where task requests and results are stored.

```bash
$> ipcontroller --sqlitedb # or --mongodb or --nodb
```

--------------------------------

### Manage IPython Parallel Cluster with Synchronous Context Manager

Source: https://ipyparallel.readthedocs.io/en/latest/examples/Cluster%20API

This example shows how to use `ipp.Cluster` as a synchronous context manager. The cluster is automatically started upon entering the `with` block and torn down upon exiting, simplifying resource management and ensuring proper cleanup.

```Python
import os

with ipp.Cluster(n=4) as rc:
    engine_pids = rc[:].apply_async(os.getpid).get_dict()
engine_pids
```

```Output
Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
Stopping engine(s): 1623757397-ng0s
Stopping controller

{0: 24989, 1: 24991, 2: 24990, 3: 24992}
```

--------------------------------

### Sample PBS Batch Script Template for IPython Controller

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This bash script provides a template for submitting the IPython controller job to a PBS cluster. It defines essential PBS job parameters and the command to launch the controller, similar to the engine template but typically for a single node.

```bash
#!/bin/bash
#PBS -N ipython
#PBS -V
#PBS -j oe
#PBS -o {output_file}
#PBS -l walltime=00:10:00
#PBS -l nodes=1:ppn=1
#PBS -q {queue}
#PBS -V

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
{program_and_args} # or ipcontroller --ip=*
```

--------------------------------

### Define Custom Dependencies with @ipp.depend

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/task.md

Shows how to use the `@ipp.depend` decorator with a custom dependency function. The dependency function is called at the start of the task, and if it returns `False`, the task is reassigned to another engine. This example uses a platform-specific check to run tasks only on macOS or Windows.

```Python
In [10]: def platform_specific(plat):
   ....:    import sys
   ....:    return sys.platform == plat

In [11]: @ipp.depend(platform_specific, 'darwin')
   ....: def mactask():
   ....:     do_mac_stuff()

In [12]: @ipp.depend(platform_specific, 'nt')
   ....: def wintask():
   ....:     do_windows_stuff()
```

--------------------------------

### PBS Controller Batch Script Template

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/process

A complete PBS batch script template for launching the IPython controller. It includes PBS directives for job naming, output, walltime, node allocation, and queue selection, along with commands to change directory and set environment variables.

```Bash
#!/bin/bash
#PBS -N ipython
#PBS -V
#PBS -j oe
#PBS -o {output_file}
#PBS -l walltime=00:10:00
#PBS -l nodes=1:ppn=1
#PBS -q {queue}
#PBS -V

cd $PBS_O_WORKDIR
export PATH=$HOME/usr/local/bin
{program_and_args} # or ipcontroller --ip=*
```

--------------------------------

### Connect to Existing IPython Parallel Cluster

Source: https://ipyparallel.readthedocs.io/en/latest/tutorial/intro

Shows various methods to connect a client to an already running IPython parallel cluster. This includes connecting from a default file, specifying a profile or a custom file path, and connecting via SSH to a remote hub.

```Python
cluster = ipp.Cluster.from_file()
```

```Python
cluster = ipp.Cluster.from_file(profile="myprofile", cluster_id="...")
rc = cluster.connect_client_sync()
```

```Python
cluster = ipp.Cluster.from_file('/path/to/my/cluster-.json')
```

```Python
c = ipp.Client('/path/to/my/ipcontroller-client.json', sshserver='me@myhub.example.com')
```

--------------------------------

### Initialize and Connect to IPython Parallel Cluster

Source: https://ipyparallel.readthedocs.io/en/latest/examples/Parallel%20Magics

This Python snippet demonstrates the initial setup for using IPython Parallel. It starts a local cluster with 4 engines, connects to it, and obtains a DirectView (`dv`) representing all connected engines. It also prints the IDs of the active engines.

```python
import ipyparallel as ipp

rc = ipp.Cluster(n=4).start_and_connect_sync()
dv = rc[:]
rc.ids
```

--------------------------------

### Creating a New IPython Parallel Profile

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

Illustrates the command-line method to create a new IPython Parallel configuration profile. This action generates a dedicated directory with default configuration files, allowing for customized cluster settings.

```bash
ipython profile create --parallel --profile=myprofile
```

--------------------------------

### Configure IPyParallel Controller Database Backend

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This Python snippet demonstrates how to specify the database backend for the IPyParallel controller by setting the `IPController.db_class` trait in `ipcontroller_config.py`. The example uses `ipyparallel.controller.dictdb.DictDB`, which is the default in-memory, dict-based implementation, known for its speed as it avoids filesystem operations.

```python
c.IPController.db_class = 'ipyparallel.controller.dictdb.DictDB'
```

--------------------------------

### Create IPython Parallel Profile for PBS

Source: https://ipyparallel.readthedocs.io/en/latest/_sources/tutorial/process.md

This command initializes a new IPython profile specifically configured for parallel computing and using the PBS batch system. It sets up the necessary directory structure and default configuration files for a PBS-managed cluster.

```bash
ipython profile create --parallel --profile=pbs
```
Tokens:
Raw