### Quickstart Coiled Installation and Setup

Source: https://docs.coiled.io/blog/how-to-run-a-python-script-on-ec2.html

A quick command sequence to install Coiled and perform an initial setup. This is a convenient way to get started with Coiled for running Python on the cloud.

```bash
$ pip install coiled
$ coiled quickstart

```

--------------------------------

### Install and Quickstart Coiled

Source: https://docs.coiled.io/blog/choosing-an-aws-batch-alternative.html

Install the Coiled package using pip and initiate a quickstart guide to begin using Coiled.

```bash
pip install coiled
coiled quickstart

```

--------------------------------

### Run Coiled Quickstart CLI

Source: https://docs.coiled.io/user_guide/getting_started.html

Execute the `coiled quickstart` command to begin with Coiled. This interactive CLI guides you through logging in, running a 'Hello world' example, and exploring larger computations.

```bash
$ coiled quickstart
```

--------------------------------

### Setup Coiled and Run a Command

Source: https://docs.coiled.io/blog/easy-serverless.html

Install Coiled, set up your cloud account connection, and then run a simple command remotely. This is a typical starting sequence for using Coiled.

```bash
pip install coiled
coiled setup  # this will connect Coiled to your cloud account
coiled run echo "Hello, world"
```

--------------------------------

### Install Coiled and Setup AWS Quotas

Source: https://docs.coiled.io/user_guide/setup/aws/reference.html

Installs the Coiled package and initiates the AWS setup process, including viewing or adjusting service quotas. This is useful when encountering 'LimitExceeded' errors.

```bash
pip3 install coiled && \
coiled setup aws --quotas

```

--------------------------------

### Install Coiled and Set Up AWS

Source: https://docs.coiled.io/blog/coiled-cloud-architecture.html

Install the Coiled library, log in to your Coiled account, and set up AWS as the cloud provider. This is the initial setup required to use Coiled.

```bash
pip install coiled
coiled login
coiled setup aws
ipython

```

--------------------------------

### Install necessary packages

Source: https://docs.coiled.io/examples/hpo.html

Install Optuna, Dask, Coiled, XGBoost, scikit-learn, and matplotlib using conda. This sets up the environment for the example.

```bash
$ conda create -n dask-optuna-example -c conda-forge python=3.10 optuna optuna-integration dask coiled xgboost scikit-learn matplotlib
$ conda activate dask-optuna-example

```

--------------------------------

### Install and Setup Coiled for GCP

Source: https://docs.coiled.io/blog/dask-gcp.html

Install the Coiled package, log in to your Coiled account, and set up GCP credentials for cluster deployment.

```bash
$ pip install coiled
$ coiled login
$ coiled setup gcp
```

--------------------------------

### Install Software with MPI Setup

Source: https://docs.coiled.io/user_guide/mpi.html

Specify Python packages with `--pip`, system packages with `--apt`, and custom setup scripts with `--setup-script` during cluster creation. Docker and package sync are not supported.

```bash
coiled mpi setup \
  --pip mpi4pi \
  --apt curl --apt tree \
  --setup-script download_and_do_stuff.sh
```

--------------------------------

### Install Coiled

Source: https://docs.coiled.io/_downloads/28004e0de36004972e522c737b1267ef/scaling-xarray-cloud-coiled.ipynb

Install the Coiled library to get started with distributed computing. This is a prerequisite for using Coiled with Xarray.

```bash
pip install coiled
```

--------------------------------

### Run Coiled Azure Setup

Source: https://docs.coiled.io/user_guide/setup/azure/cli.html

Execute this command to initiate the Coiled setup process for Azure. It will guide you through creating a service principal with appropriate permissions for your Azure resource group.

```bash
coiled setup azure

```

--------------------------------

### Coiled Azure Setup Command Reference

Source: https://docs.coiled.io/user_guide/setup/azure/cli.html

This shows the basic structure of the `coiled setup azure` command and its available options for customizing the setup process.

```bash
coiled setup azure [OPTIONS]

```

--------------------------------

### Start a Coiled Notebook with File Sync

Source: https://docs.coiled.io/user_guide/notebooks.html

Use the `--sync` flag with `coiled notebook start` to synchronize local notebook files with the remote machine. This requires Mutagen to be installed.

```bash
# synchronize changes between cloud and local
coiled notebook start --sync
```

--------------------------------

### Install Coiled and Xarray

Source: https://docs.coiled.io/_downloads/28004e0de36004972e522c737b1267ef/scaling-xarray-cloud-coiled.ipynb

Install the necessary libraries for using Coiled with Xarray. This is a prerequisite for all subsequent examples.

```bash
pip install "coiled[xarray]"
```

--------------------------------

### Install Prefect and Coiled

Source: https://docs.coiled.io/user_guide/prefect.html

Install the necessary Python packages for Prefect and Coiled integration.

```bash
pip install prefect coiled prefect-coiled

```

--------------------------------

### Start a Coiled Notebook with Specific VM Type

Source: https://docs.coiled.io/user_guide/notebooks.html

Configure the VM type for your Coiled notebook server using the `--vm-type` flag. Examples include specifying a high-core count machine or a GPU-enabled instance.

```bash
# 128 Cores and 512 GB of RAM
coiled notebook start --vm-type m6i.32xlarge
```

```bash
# A single T4 NVIDIA GPU
coiled notebook start --vm-type g4dn.xlarge
```

--------------------------------

### Install dask-jobqueue

Source: https://docs.coiled.io/blog/xarray-at-scale.html

Install the `dask-jobqueue` library to manage Dask clusters on HPC environments with job schedulers.

```bash
pip install dask-jobqueue
```

--------------------------------

### Install Coiled and PyTorch with Pip

Source: https://docs.coiled.io/blog/aws-lambda-gpu-alternative.html

Use pip to install Coiled and PyTorch for GPU-accelerated workloads.

```bash
$ pip install torch torchvision coiled
```

--------------------------------

### Install Coiled Notebooks with Pip

Source: https://docs.coiled.io/blog/jupyter-notebook-gpu.html

Install the Coiled package with notebook dependencies and PyTorch using pip.

```bash
$ pip install "coiled[notebook]" torch torchvision
```

--------------------------------

### Scatter plot with Datashader aggregation

Source: https://docs.coiled.io/_downloads/dd8dcb2da96b0d8d1ee59708d946a7a0/datashader.ipynb

Visualize large datasets by creating a scatter plot with Datashader aggregation. This example uses hvplot to generate the plot, specifying aggregation by 'journey_type' and enabling datashading with an 'eq_hist' normalization. Color mapping is applied for 'pickup' and 'dropoff' categories.

```python
import datashader
import holoviews as hv
import hvplot.dask
hv.extension('bokeh')

df.hvplot.scatter(
    x="lon", 
    y="lat", 
    aggregator=datashader.by("journey_type"), 
    datashade=True, 
    cnorm="eq_hist",
    frame_width=700, 
    aspect=1.33, 
    color_key={"pickup": "#EF1561", "dropoff": "#1F5AFF"},
)
```

--------------------------------

### Coiled GCP Setup Command Help

Source: https://docs.coiled.io/user_guide/setup/gcp/cli.html

Display the help information for the `coiled setup gcp` command, including available options for configuration.

```bash
coiled setup gcp [OPTIONS]

```

--------------------------------

### Failed Dask Scheduler Verification Output

Source: https://docs.coiled.io/user_guide/software/docker.html

Example output when the Dask scheduler fails to start due to a missing 'distributed' module. This indicates a problem with the container's environment setup.

```text
> docker run --rm continuumio/miniconda3:latest python -m distributed.cli.dask_spec \
    --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

Unable to find image 'continuumio/miniconda3:latest' locally
latest: Pulling from continuumio/miniconda3
dc1f00a5d701: Already exists
a7a9c78d89b2: Already exists
44ac19016d77: Already exists
Digest: sha256:977263e8d1e476972fddab1c75fe050dd3cd17626390e874448bd92721fd659b
Status: Downloaded newer image for continuumio/miniconda3:latest
/opt/conda/bin/python: Error while finding module specification for 'distributed.cli.dask_spec' (ModuleNotFoundError: No module named 'distributed')

```

--------------------------------

### Setup and Run Basic MPI Hello World

Source: https://docs.coiled.io/user_guide/mpi.html

Use `coiled mpi setup` to create an MPI-enabled cluster and `coiled mpi run` to execute commands. The cluster automatically shuts down after an idle timeout.

```bash
coiled mpi setup  # create the cluster (this takes 4 to 5 minutes)

coiled mpi run -- echo hello  # this will print many lines of "hello"
```

--------------------------------

### AWS Setup with CloudShell Link

Source: https://docs.coiled.io/user_guide/setup/aws/cli.html

Generate setup instructions specifically for AWS CloudShell, which is useful if you don't have local AWS credentials configured.

```bash
coiled setup aws --cloudshell-link

```

--------------------------------

### Install Coiled and Set Up Azure

Source: https://docs.coiled.io/blog/managed-dask.html

Install the Coiled library and log in to your Coiled account. Then, set up Coiled to manage Dask clusters on Azure.

```bash
$ pip install coiled
$ coiled login
$ coiled setup azure
```

--------------------------------

### Install Packages with Conda

Source: https://docs.coiled.io/blog/coiled-run-duckdb.html

Install necessary Python packages for Coiled and DuckDB using Conda. This sets up a virtual environment for the example.

```bash
conda create -n coiled-duckdb-example -c conda-forge python=3.10 coiled s3fs python-duckdb jupyterlab
conda activate coiled-duckdb-example

```

--------------------------------

### Install Coiled and Login

Source: https://docs.coiled.io/user_guide/setup/web-app.html

Install the Coiled client and authenticate your machine with the Coiled service. This process saves an API token for future use.

```bash
pip install coiled "dask[complete]"
coiled login

```

--------------------------------

### Install Packages with Conda

Source: https://docs.coiled.io/_downloads/b9cea90e4d460bf15c7b54a549e67951/hpo.ipynb

Installs necessary packages for the Dask-Optuna example using Conda. This ensures the environment is replicated on the Dask cluster.

```bash
$ conda create -n dask-optuna-example -c conda-forge python=3.10 optuna optuna-integration dask coiled xgboost scikit-learn matplotlib
$ conda activate dask-optuna-example
```

--------------------------------

### Install Coiled and Set Up Cloud Credentials

Source: https://docs.coiled.io/blog/dask-azure.html

Install the Coiled package and log in to your Coiled account. Then, set up cloud provider credentials, such as AWS, to enable Coiled to provision resources.

```bash
$ pip install coiled
$ coiled login
$ coiled setup aws
```

--------------------------------

### Install Packages with Conda

Source: https://docs.coiled.io/examples/datashader.html

Installs necessary packages including Coiled, Dask, and Datashader in a new virtual environment. This setup is automatically replicated on the Coiled cluster.

```bash
conda create -n coiled-datashader -c conda-forge python=3.10 coiled dask s3fs pyarrow datashader hvplot jupyter_bokeh
conda activate coiled-datashader

```

--------------------------------

### Example of an Unsolvable Environment

Source: https://docs.coiled.io/user_guide/software/package_sync_limitations.html

This example shows an environment configuration that will fail during installation because of incompatible version requirements between dask and distributed. Ensure your package versions are compatible.

```yaml
dask==2022.1.10
distributed==2022.05.1
```

--------------------------------

### Install and Login to Coiled

Source: https://docs.coiled.io/blog/how-to-run-pandas-on-gpus-with-coiled.html

Install the Coiled library and log in to authenticate your machine. This process redirects to the Coiled website for authentication and saves a token locally.

```bash
pip install coiled
coiled login
```

--------------------------------

### Start a Coiled Notebook

Source: https://docs.coiled.io/blog/coiled-notebooks.html

Launch a Coiled notebook to provision a cloud instance, set up a JupyterLab session, and connect to it in your browser.

```bash
coiled notebook start
```

--------------------------------

### Example environment.yml for Conda

Source: https://docs.coiled.io/blog/just-in-time-python-environments.html

This is an example environment.yml file used for defining Python dependencies and configurations for Conda environments. It specifies channels and a list of packages to be installed.

```yaml
name: base
channels:
  - conda-forge
  - defaults
dependencies:
  - gcc
  - _libgcc_mutex=0.1=conda_forge
  -_openmp_mutex=4.5=2_gnu
  - abseil-cpp=20210324.2=h9c3ff4c_0
  - aiobotocore=2.3.4=pyhd8ed1ab_0
  - aiohttp=3.8.1=py39hb9d737c_1
  - aioitertools=0.10.0=pyhd8ed1ab_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - alembic=1.8.1=pyhd8ed1ab_0
  - alsa-lib=1.2.6.1=h7f98852_0
  - anyio=3.6.1=py39hf3d152e_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - argon2-cffi=21.3.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py39hb9d737c_2
  - arrow-cpp=8.0.0=py39h811ffd7_4_cpu
  - asn1crypto=1.5.1=pyhd8ed1ab_0
  - asttokens=2.0.5=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - attr=2.5.1=h166bdaf_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - aws-c-cal=0.5.11=h95a6274_0
  - aws-c-common=0.6.2=h7f98852_0
  - aws-c-event-stream=0.2.7=h3541f99_13
  - aws-c-io=0.10.5=hfb6a706_0
  - aws-checksums=0.1.11=ha31a3da_7
  - aws-sdk-cpp=1.8.186=hb4091e7_3
  - babel=2.10.3=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - beautifulsoup4=4.11.1=pyha770c72_0
  - bleach=5.0.1=pyhd8ed1ab_0
  - blinker=1.4=py_1
  - blosc=1.21.1=h83bc5f7_3
  - bokeh=2.4.3=py39hf3d152e_0
  - botocore=1.24.21=pyhd8ed1ab_1
  - brotli=1.0.9=h166bdaf_7
  - brotli-bin=1.0.9=h166bdaf_7
  - brotlipy=0.7.0=py39hb9d737c_1004
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.6.15=ha878542_0
  - certifi=2022.6.15=py39hf3d152e_0
  - cffi=1.15.1=py39he91dace_0
  - click=8.1.3=py39hf3d152e_0
  - cloudpickle=2.1.0=pyhd8ed1ab_0
  - colorama=0.4.5=pyhd8ed1ab_0
  - configparser=5.2.0=pyhd8ed1ab_0
  - cryptography=37.0.4=py39hd97740a_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - cytoolz=0.12.0=py39hb9d737c_0
  - dask=2022.7.1=pyhd8ed1ab_0
  - dask-core=2022.7.1=pyhd8ed1ab_0
  - databricks-cli=0.17.0=pyhd8ed1ab_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.6.0=py39h5a03fae_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - distributed=2022.7.1=pyhd8ed1ab_0
  - docker-py=5.0.3=py39hf3d152e_2
  - docker-pycreds=0.4.0=py_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - executing=0.9.0=pyhd8ed1ab_0
  - expat=2.4.8=h27087fc_0
  - fftw=3.3.10=nompi_h77c792f_102
  - flask=2.1.3=pyhd8ed1ab_0
  - flit-core=3.7.1=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.14.0=h8e229c2_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.34.4=py39hb9d737c_0
  - freetype=2.10.4=h0708190_1
  - frozenlist=1.3.0=py39hb9d737c_1
  - fsspec=2022.5.0=pyhd8ed1ab_0
  - gettext=0.19.8.1=h73d1719_1008
  - gflags=2.2.2=he1b5a44_1004
  - giflib=5.2.1=h36c2ea0_2
  - gitdb=4.0.9=pyhd8ed1ab_0
  - gitpython=3.1.27=pyhd8ed1ab_0
  - glib=2.72.1=h6239696_0
  - glib-tools=2.72.1=h6239696_0
  - glog=0.6.0=h6f12383_0
  - greenlet=1.1.2=py39h5a03fae_2
  - grpc-cpp=1.45.2=h3b8df00_4
  - gst-plugins-base=1.20.3=hf6a322e_0
  - gstreamer=1.20.3=hd4edc92_0
  - gunicorn=20.1.0=py39hf3d152e_2
  - heapdict=1.0.1=py_0
  - htmlmin=0.1.12=py_1
  - icu=70.1=h27087fc_0
  - idna=3.3=pyhd8ed1ab_0
  - imagehash=4.2.1=pyhd8ed1ab_0
  - importlib-metadata=4.11.4=py39hf3d152e_0
  - importlib_metadata=4.11.4=hd8ed1ab_0
  - importlib_resources=5.9.0=pyhd8ed1ab_0
  - ipykernel=6.15.1=pyh210e3f2_0
  - ipython=8.4.0=py39hf3d152e_0
  - ipython_genutils=0.2.0=py_1
  - itsdangerous=2.1.2=pyhd8ed1ab_0
  - jack=1.9.18=h8c3723f_1002
  - jedi=0.18.1=py39hf3d152e_1
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jmespath=1.0.1=pyhd8ed1ab_0
  - joblib=1.1.0=pyhd8ed1ab_0
  - jpeg=9e=h166bdaf_2
  - json5=0.9.5=pyh9f0ad1d_0
  - jsonschema=4.7.2=pyhd8ed1ab_0
  - jupyter_client=7.3.4=pyhd8ed1ab_0
  - jupyter_core=4.11.1=py39hf3d152e_0
  - jupyter_server=1.18.1=pyhd8ed1ab_0
  - jupyterlab=3.4.4=pyhd8ed1ab_0
  - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
  - jupyterlab_server=2.15.0=pyhd8ed1ab_0
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.4=py39hf939315_0
  - krb5=1.19.3=h3790be6_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libblas=3.9.0=15_linux64_openblas
  - libbrotlicommon=1.0.9=h166bdaf_7

```

--------------------------------

### Configure Coiled notebook start with zone and capacity reservation

Source: https://docs.coiled.io/user_guide/changelog.html

Add `--zone`, `--capacity-reservation-id`, and `--capacity-reservation-group-arn` to `coiled notebook start` CLI for advanced instance provisioning.

```bash
coiled notebook start --zone us-east-1 --capacity-reservation-id my-cr-id --capacity-reservation-group-arn my-cr-group-arn ...
```

--------------------------------

### Start a Coiled Notebook

Source: https://docs.coiled.io/user_guide/notebooks.html

Replace your local Jupyter command with `coiled notebook start` to launch a notebook server in the cloud. This process takes 1-2 minutes and automatically opens a browser tab.

```bash
# jupyter lab            # replace this
coiled notebook start    # with this
```

--------------------------------

### Start a Coiled Notebook Instance

Source: https://docs.coiled.io/examples/ml.html

Launch a Jupyter Notebook on a cloud instance with specified VM type and region using the `coiled notebook start` command. Use `--sync` to synchronize local files.

```bash
$ coiled notebook start \
    --vm-type g5.8xlarge \
    --region us-west-2 \
    --sync                   

```

--------------------------------

### Coiled AWS Setup Command Reference

Source: https://docs.coiled.io/user_guide/setup/aws/cli.html

The main command for setting up AWS with Coiled, showing its available options for configuration.

```bash
coiled setup aws [OPTIONS]

```

--------------------------------

### Start Cloud Dask Cluster

Source: https://docs.coiled.io/user_guide/dask.html

Initiates a Dask cluster on the cloud using Coiled. Requires `coiled` to be installed.

```python
from coiled import Cluster

cluster = Cluster()
client = cluster.get_client()

```

--------------------------------

### Successful Dask Scheduler Verification Output

Source: https://docs.coiled.io/user_guide/software/docker.html

Example output when the Dask scheduler successfully starts within a Docker container during verification.

```text
> docker run --rm daskdev/dask:latest python -m distributed.cli.dask_spec \
    --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

2022-10-06 14:44:43,640 - distributed.scheduler - INFO - State start
2022-10-06 14:44:43,656 - distributed.scheduler - INFO - Clear task state
2022-10-06 14:44:43,658 - distributed.scheduler - INFO -   Scheduler at:    tcp://172.17.0.2:41089
2022-10-06 14:44:43,658 - distributed.scheduler - INFO -   dashboard at:                     :8787

```

--------------------------------

### Get Spark Client

Source: https://docs.coiled.io/_modules/coiled/v2/cluster.html

Obtain a Spark client connected to the Coiled cluster. This is an experimental feature. Ensure the cluster was started with `coiled.spark.get_spark_cluster`.

```python
from coiled.spark import SPARK_CONNECT_PORT, get_spark

self._spark_dashboard = parse_url(self._dashboard_address)._replace(path="/spark").url
self._spark_master = parse_url(self._dashboard_address)._replace(path="/spark-master").url
dashboards = (
    "\n"
    f"[bold green]Spark UI:[/]	 [link={self._spark_dashboard}]{self._spark_dashboard}[/link]"
    "\n\n"
    f"[bold green]Spark Master:[/]	 [link={self._spark_master}]{self._spark_master}[/link]"
    "\n"
)

if self.use_dashboard_https:
    host = parse_url(self._dashboard_address).host
    token = parse_url(self._dashboard_address).query
    remote_address = f"sc://{host}:{SPARK_CONNECT_PORT}/";use_ssl=true;{token}"
else:
    remote_address = None

with self.get_client() as client:
    spark_session = get_spark(
        client,
        connection_string=remote_address,
        block_till_ready=block_till_ready,
        spark_connect_config=spark_connect_config,
        executor_memory_factor=executor_memory_factor,
        worker_memory_factor=worker_memory_factor,
    )
    if self._spark_dashboard.startswith("https"):
```

--------------------------------

### Start a notebook with Deep Learning AMI using coiled notebook start

Source: https://docs.coiled.io/user_guide/changelog.html

Use the `--ami-version DL` option to specify the Deep Learning Base OSS Nvidia Driver GPU AMI for Coiled notebooks.

```bash
coiled notebook start --ami-version DL ...
```

--------------------------------

### List Local Coiled Versions

Source: https://docs.coiled.io/user_guide/api.html

Get information about the locally installed versions of Python, Coiled, Dask, and Distributed. Useful for troubleshooting. Can return output as JSON.

```python
coiled.list_local_versions()
```

--------------------------------

### Start a GPU-enabled Jupyter Notebook with Coiled

Source: https://docs.coiled.io/blog/how-to-run-pandas-on-gpus-with-coiled.html

Launches a JupyterLab instance on a cloud VM with GPU hardware. Coiled handles provisioning, software environment setup, and teardown.

```bash
coiled notebook start --gpu --software rapids
```

--------------------------------

### Install Google Cloud SDK and Login

Source: https://docs.coiled.io/user_guide/data/remote-data-access.html

Install the Google Cloud SDK and log in to configure your local Application Default Credentials. This is a prerequisite for both OAuth2 token generation and forwarding ADC.

```bash
conda install -c conda-forge google-cloud-sdk
gcloud auth application-default login

```

--------------------------------

### Get Spark Client

Source: https://docs.coiled.io/_modules/coiled/v2/cluster.html

Retrieves a Spark client connected to the Coiled cluster. This is an experimental feature and may change without notice. It requires the cluster to be started with `coiled.spark.get_spark_cluster`.

```APIDOC
## get_spark

### Description
Retrieves a Spark client connected to the Coiled cluster. This is an experimental feature and may change without notice. It requires the cluster to be started with `coiled.spark.get_spark_cluster`.

### Parameters
- **block_till_ready** (bool) - Optional - Defaults to True. Whether to block until the Spark client is ready.
- **spark_connect_config** (dict | None) - Optional - Dictionary of additional config options for Spark Connect.
- **executor_memory_factor** (float | None) - Optional - Factor to determine `spark.executor.memory` based on available memory (between 0 and 1).
- **worker_memory_factor** (float | None) - Optional - Factor to determine `--memory` for `org.apache.spark.deploy.worker.Worker` (between 0 and 1).

### Request Example
```python
spark_client = cluster.get_spark()
spark_client = cluster.get_spark(spark_connect_config={"spark.foo": "123"})
```

### Response
- **spark_session** (object) - A Spark session object that can be used to interact with Spark.

### Additional Information
- The Spark UI and Master URLs are printed to the console upon successful retrieval of the Spark client.
```

--------------------------------

### Get Cluster Aggregated Metric

Source: https://docs.coiled.io/_modules/coiled/v2/core.html

Asynchronously retrieves aggregated metrics for a cluster, allowing for custom queries and time ranges. Supports optional start and end timestamps.

```python
async def _get_cluster_aggregated_metric(
        self,
        cluster_id: int,
        workspace: str | None,
        query: str,
        over_time: str,
        start_ts: int | None,
        end_ts: int | None,
    ):
        workspace = workspace or self.default_workspace
        route = f"/api/v2/metrics/account/{workspace}/cluster/{cluster_id}/value"
        url = f"{self.server}{route}"
        params = {"query": query, "over_time": over_time}
        if start_ts:
            params["start_ts"] = str(start_ts)
        if end_ts:
            params["end_ts"] = str(end_ts)

        response = await self._do_request("GET", url, params=params)
        if response.status >= 400:
            await handle_api_exception(response)
        return await response.json()
```

--------------------------------

### Set up Python environment on Ubuntu

Source: https://docs.coiled.io/blog/how-to-run-a-python-script-on-ec2.html

Update the system, install Python 3 and venv, and activate a virtual environment on an Ubuntu EC2 instance.

```bash
# Update the system
sudo apt update && sudo apt upgrade -y

# Install Python
sudo apt install python3 python3-venv -y

# Set up a virtual environment
python3 -m venv env
source env/bin/activate

```

--------------------------------

### Get AWS Default Region

Source: https://docs.coiled.io/_modules/coiled/v2/cluster.html

Retrieves the default AWS region configured for the environment. It attempts to use boto3 to find the region and returns None if it cannot be determined or if boto3 is not installed.

```python
@staticmethod
def _get_aws_default_region() -> str | None:
        try:
            from boto3.session import Session

            region_name = Session().region_name
            return str(region_name) if region_name else None
        except Exception:
            pass
        return None
```

--------------------------------

### Run Data Pipeline Locally

Source: https://docs.coiled.io/blog/coiled-from-prefect.html

Clone the example GitHub repository, set up the environment, and run the Prefect workflow locally to test the data pipeline.

```bash
# Run data pipeline locally
git clone https://github.com/coiled/etl-tpch
cd etl-tpch
mamba env create -f environment.yml
mamba activate etl-tpch
python workflow.py

```

--------------------------------

### Install Coiled Client and Login

Source: https://docs.coiled.io/user_guide/getting_started.html

Install the Coiled Python library and log in to authenticate your computer. This command will redirect you to the Coiled website and save your API token.

```bash
pip install coiled "dask[complete]"
coiled login
```

--------------------------------

### Get Dask DataFrame Divisions

Source: https://docs.coiled.io/blog/dask-filter-dataframes-loc.html

Retrieve and print the division information for a Dask DataFrame. This metadata indicates the start and end index values for each partition, crucial for efficient filtering.

```python
ddf.divisions(0, 3, 5)
```

--------------------------------

### Setup MPI Cluster with cuPyNumeric

Source: https://docs.coiled.io/user_guide/mpi.html

Install `nvidia-cupynumeric` using `--pip` and specify the number of worker nodes. The `--legate` flag enables distributed cuPyNumeric work across nodes.

```bash
coiled mpi setup --pip nvidia-cupynumeric --worker-nodes 2
```

--------------------------------

### Configure Firewall and Backend Options

Source: https://docs.coiled.io/_modules/coiled/core.html

Sets up firewall and backend options, with a warning for deprecated `firewall` usage. It prepares backend configuration for AWS or GCP.

```python
    if firewall:
        if ingress:
            raise ValueError(
                "You specified both `firewall` and `ingress`. These are redundant; you should use `ingress`."
            )
        else:
            logger.warning(
                "The `firewall` keyword argument is deprecated; in the future you should use\n"
                f"  ingress=[{{ {firewall} }}]\n"
                "to specify your desired firewall ingress rules."
            )
            ingress = [firewall]
    firewall_spec = {"ingress": ingress} if ingress else {}

    # TODO - see if way to add default in BE to avoid re-versioning of this
    backend_options = {
        "backend": "vm_aws",
        "options": {
            "aws_region_name": aws_region,
            "account_role": "",
            "credentials": {"aws_access_key_id": "", "aws_secret_access_key": ""},
            "firewall": {},
            "firewall_spec": firewall_spec,
            "network": network or {},
            "zone": zone,
        },
        "registry": {"type": "ecr", "credentials": {}, "public_ecr": False},
    }

    # override gcp_zone with zone, if set
    if zone and gcp_project_id:
        gcp_zone = zone

    output_msg = ""
    # Used to print warnings to the user
    console = Console()

    if backend not in SUPPORTED_BACKENDS:
        raise UnsupportedBackendError(f"Supplied backend: {backend} not in supported types: {SUPPORTED_BACKENDS}")

    if aws_access_key_id and aws_secret_access_key:
        # verify that these are valid credentials
        verify_aws_credentials_with_retry(aws_access_key_id, aws_secret_access_key)

    parsed_gcp_credentials: Dict | None = None

    # Parse GCP region/zones or return default region/zone
    gcp_region, gcp_zone = parse_gcp_region_zone(region=gcp_region, zone=gcp_zone)

    if backend == "aws":
        backend_options["backend"] = "vm"
        backend_options["options"]["aws_region_name"] = aws_region
        if aws_access_key_id and aws_secret_access_key:
            backend_options["options"]["credentials"] = {
                "aws_access_key": aws_access_key_id,
                "aws_secret_key": aws_secret_access_key,
            }

            backend_options["options"]["provider_name"] = "aws"
            backend_options["options"]["type"] = "aws_cloudbridge_backend_options"
            output_msg = "Successfully set your backend options to Coiled Customer Hosted on AWS VM."
        else:
            raise AWSCredentialsParameterError(
                "Setting up AWS backend requires both: aws_access_key_id and aws_secret_access_key."
            )
    elif backend == "gcp":
        parsed_gcp_credentials = _parse_gcp_creds(
            gcp_service_creds_dict=gcp_service_creds_dict,
            gcp_service_creds_file=gcp_service_creds_file,
        )
        if not gcp_project_id:
            gcp_project_id = parsed_gcp_credentials.get("project_id", "")
        backend_options["options"]["gcp_service_creds_dict"] = parsed_gcp_credentials

        backend_options["backend"] = "vm"

```