### Example of Symlinking Certificates for Arvados Services

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Demonstrates how to create symbolic links for certificate and key files for various Arvados services when using a single certificate file.

```bash
ln -s xarv1.crt ${CUSTOM_CERTS_DIR}/controller.crt
ln -s xarv1.key ${CUSTOM_CERTS_DIR}/controller.key
ln -s xarv1.crt ${CUSTOM_CERTS_DIR}/keepproxy.crt
ln -s xarv1.key ${CUSTOM_CERTS_DIR}/keepproxy.key
```

--------------------------------

### Start Arvados Services

Source: https://doc.arvados.org/v3.1/install/install-api-server

Command to enable and start the Arvados API server and controller services.

```APIDOC
## Start the service

```bash
# systemctl enable --now arvados-railsapi arvados-controller
```
```

--------------------------------

### Enable and Start Arvados Services

Source: https://doc.arvados.org/v3.1/install/install-api-server

Enables and starts the Arvados Rails API server and Arvados controller services using systemctl. This ensures the services start automatically on boot and are running immediately.

```bash
# systemctl enable --now arvados-railsapi arvados-controller
```

--------------------------------

### Install arvados-client

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Installs the arvados-client package using apt, which is required for running Arvados diagnostics.

```bash
apt install arvados-client

```

--------------------------------

### Download Arvados Installer with Git

Source: https://doc.arvados.org/v3.1/install/salt-single-host

This code block demonstrates how to download the Arvados installation files using git. It clones the repository, checks out the specific release tag (3.1-release), and navigates to the salt-install tools directory.

```shell
git clone https://github.com/arvados/arvados.git
cd arvados
git checkout 3.1-release
cd tools/salt-install
```

--------------------------------

### Initialize Arvados Installation (Shell)

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Initializes the Arvados installation process using a shell script. Sets up the cluster with a specified configuration mode (e.g., single_host_single_hostname). Navigate to the created directory to proceed.

```shell
CLUSTER=xarv1
./installer.sh initialize ~/setup-arvados-${CLUSTER} single_host_single_hostname single_host/single_hostname
cd ~/setup-arvados-${CLUSTER}
```

--------------------------------

### Initialize Arvados Installer with Terraform on AWS

Source: https://doc.arvados.org/v3.1/install/salt-single-host

This command initializes the Arvados installer using a cluster ID and Terraform for a single-host, single-hostname setup on AWS. It creates a setup directory and navigates into it, preparing for further configuration.

```shell
CLUSTER=xarv1
./installer.sh initialize ~/setup-arvados-${CLUSTER} single_host_single_hostname single_host/single_hostname 
cd ~/setup-arvados-${CLUSTER}
```

--------------------------------

### Manage keepstore service with systemctl

Source: https://doc.arvados.org/v3.1/install/install-keepstore

Commands to enable, start, and check the status of the keepstore service. Also includes checking logs for errors using journalctl.

```bash
# systemctl enable --now keepstore
# systemctl status keepstore

```

```bash
# journalctl --since -5min -u keepstore

```

--------------------------------

### Deploy Arvados Configuration

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Deploys Arvados configuration changes, including database setup and synchronization of Arvados services. This script is essential after initial installation or configuration updates.

```bash
./installer.sh deploy
```

--------------------------------

### Terraform Output for Arvados Cluster

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Example output from the Arvados Terraform installer, detailing created AWS resources like security group IDs, subnet IDs, domain names, and IP addresses for various cluster services.

```bash
./installer.sh terraform
...
Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

Outputs:

arvados_sg_id = "sg-02f999a99973999d7"
arvados_subnet_id = "subnet-01234567abc"
cluster_int_cidr = "10.1.0.0/16"
cluster_name = "xarv1"
compute_subnet_id = "subnet-abcdef12345"
deploy_user = "admin"
domain_name = "xarv1.example.com"
letsencrypt_iam_access_key_id = "AKAA43MAAAWAKAADAASD"
loki_iam_access_key_id = "AKAABCDEFGJKLMNOP1234"
private_ip = {
  "controller" = "10.1.1.1"
  "keep0" = "10.1.1.3"
  "keep1" = "10.1.1.4"
  "keepproxy" = "10.1.1.2"
  "shell" = "10.1.1.7"
  "workbench" = "10.1.1.5"
}
public_ip = {
  "controller" = "18.235.116.23"
  "keep0" = "34.202.85.86"
  "keep1" = "38.22.123.98"
  "keepproxy" = "34.231.9.201"
  "shell" = "44.208.155.240"
  "workbench" = "52.204.134.136"
}
region_name = "us-east-1"
route53_dns_ns = tolist([
  "ns-1119.awsdns-11.org",
  "ns-1812.awsdns-34.co.uk",
  "ns-437.awsdns-54.com",
  "ns-809.awsdns-37.net",
])
ssl_password_secret_name = "xarv1-arvados-ssl-privkey-password"
vpc_id = "vpc-0999994998399923a"
letsencrypt_iam_secret_access_key = "XXXXXSECRETACCESSKEYXXXX"
database_password = <not set>
loki_iam_secret_access_key = "YYYYYYSECRETACCESSKEYYYYYYY"


```

--------------------------------

### Sample Slurm Configuration for Arvados

Source: https://doc.arvados.org/v3.1/install/crunch2-slurm/configure-slurm

This snippet provides an example of a `slurm.conf` file tailored for use with Arvados. It specifies cluster settings, node definitions, and partition configurations necessary for dispatching Arvados containers to a Slurm cluster. Ensure Slurm and munge are running before applying this configuration.

```slurm
ControlMachine=ClusterID.example.com
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/tmp
SlurmdSpoolDir=/tmp/slurmd
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/pgid
CacheGroups=0
ReturnToService=2
TaskPlugin=task/affinity
#
# TIMERS
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
#
# SCHEDULING
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/linear
FastSchedule=0
#
# LOGGING
SlurmctldDebug=3
#SlurmctldLogFile=
SlurmdDebug=3
#SlurmdLogFile=
JobCompType=jobcomp/none
#JobCompLoc=
JobAcctGatherType=jobacct_gather/none
#
# COMPUTE NODES
NodeName=DEFAULT
PartitionName=DEFAULT MaxTime=INFINITE State=UP

NodeName=compute[0-255]
PartitionName=compute Nodes=compute[0-255] Default=YES Shared=YES

```

--------------------------------

### Start and Enable keep-balance Service

Source: https://doc.arvados.org/v3.1/install/install-keep-balance

These systemctl commands enable the keep-balance service to start automatically on boot and then start the service immediately. It also shows how to check the service's status.

```bash
# systemctl enable --now keep-balance
# systemctl status keep-balance
```

--------------------------------

### Install Arvados API Server and Controller

Source: https://doc.arvados.org/v3.1/install/install-api-server

Package installation commands for Arvados API server and controller on different Linux distributions.

```APIDOC
## Install arvados-api-server and arvados-controller

### Red Hat, AlmaLinux, and Rocky Linux
```bash
# dnf install arvados-api-server arvados-controller
```

### Debian and Ubuntu
```bash
# apt install arvados-api-server arvados-controller
```
```

--------------------------------

### Test Arvados Python Client installation from PyPI

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Tests the Arvados Python SDK installation from PyPI by creating an API client object. Requires ARVADOS_API_HOST and ARVADOS_API_TOKEN to be set.

```python
>>> import arvados
>>> arvados.api('v1')
<apiclient.discovery.Resource object at 0x233bb50>

```

--------------------------------

### Enable and Start systemd Services (Red Hat)

Source: https://doc.arvados.org/v3.1/admin/upgrading

On Red Hat-based systems using systemd, unit files are installed but services require explicit enabling and starting. This command sequence demonstrates how to enable and start the 'keep-web' service.

```bash
sudo systemctl enable keep-web; sudo systemctl start keep-web
```

--------------------------------

### Manage Arvados Websocket Service (Systemd)

Source: https://doc.arvados.org/v3.1/install/install-ws

Enables and starts the arvados-ws service using systemd, and shows how to check its status and view logs.

```shell
# systemctl enable --now arvados-ws
# systemctl status arvados-ws

```

```shell
# journalctl --since -5min -u arvados-ws

```

--------------------------------

### Enable Development Repository for Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/install/install-api-server

Instructions on how to enable the development repository for specific Red Hat-based distributions to install Arvados packages.

```APIDOC
## Enable development repository

On Red Hat, AlmaLinux, and Rocky Linux, the API server package depends on development headers available from a separate repository. The repository you need depends on which version of the distribution you’re running. Run the command given for your distribution below:

**Distribution and version** | **Command to enable repository**  
---|---
Red Hat/AlmaLinux/Rocky Linux 9 | `# dnf config-manager --set-enabled devel`  
Red Hat/AlmaLinux/Rocky Linux 8 | `# dnf config-manager --set-enabled powertools`  
```

--------------------------------

### Arv CLI Basic Usage Examples

Source: https://doc.arvados.org/v3.1/sdk/cli/reference

Demonstrates basic commands for interacting with Arvados resources using the 'arv' CLI. Includes examples for getting user and group information, updating user records, deleting groups, and creating collections.

```bash
arv --help
arv user --help
arv user current
arv user get --uuid 6dnxa-tpzed-iimd25zhzh84gbk
arv user update --uuid 6dnxa-tpzed-iimd25zhzh84gbk --user '{"first_name":"Bob"}'
arv group list
arv group delete --uuid 6dnxa-j7d0g-iw7i6n43d37jtog
arv collection create --collection '{"name": "test collection"}'
```

--------------------------------

### Activate Arvados Python SDK virtual environment and test API client

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Activates the virtual environment for the Arvados Python SDK installed from a distribution package and demonstrates creating an API client object. Requires ARVADOS_API_HOST and ARVADOS_API_TOKEN to be set.

```bash
~$ source /usr/lib/python3-arvados-python-client/bin/activate
(python-arvados-python-client) ~$ python
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import arvados
>>> arvados.api('v1')
<apiclient.discovery.Resource object at 0x233bb50>

```

```bash
~$ /usr/lib/python3-arvados-python-client/bin/python
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import arvados
>>> arvados.api('v1')
<apiclient.discovery.Resource object at 0x233bb50>

```

--------------------------------

### Install Arvados Python Client from PyPI

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Installs the Arvados Python client using pip, recommended for use within Python programs. It can be installed into a virtual environment. Includes a workaround for older pip versions.

```python
python3 -m pip install arvados-python-client

```

```python
python3 -m pip install --pre arvados-python-client

```

--------------------------------

### Install Python SDK on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Installs the Python SDK using the apt package manager on Debian-based systems. This method makes the CLI tools available system-wide.

```bash
# apt install python3-arvados-python-client

```

--------------------------------

### Install Arvados API Server and Controller on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/install/install-api-server

Installs the Arvados API server and controller packages using the APT package manager on Debian or Ubuntu systems. This command assumes Arvados repositories are already configured.

```bash
# apt install arvados-api-server arvados-controller
```

--------------------------------

### Manage keep-web Service with systemctl

Source: https://doc.arvados.org/v3.1/install/install-keep-web

Enables and starts the keep-web service using systemctl, and provides commands to check its status and view logs for troubleshooting.

```bash
# systemctl enable --now keep-web
# systemctl status keep-web
# journalctl --since -5min -u keep-web
```

--------------------------------

### Install Python SDK on Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Installs the Python SDK using the dnf package manager on Red Hat-based systems. This method makes the CLI tools available system-wide.

```bash
# dnf install python3-arvados-python-client

```

--------------------------------

### Install Arvados Python Client dependencies on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/sdk/python/sdk-python

Installs necessary development packages for building the Arvados Python client from PyPI on Debian-based systems. This includes git, build tools, Python development headers, and SSL/libcurl libraries.

```bash
# apt install git build-essential python3-dev libcurl4-openssl-dev libssl-dev

```

--------------------------------

### Manage crunch-dispatch-slurm service with systemctl

Source: https://doc.arvados.org/v3.1/install/crunch2-slurm/install-dispatch

Commands to enable and start the crunch-dispatch-slurm service, and check its current status. These are essential for ensuring the service is running correctly.

```bash
# systemctl enable --now crunch-dispatch-slurm
# systemctl status crunch-dispatch-slurm
```

--------------------------------

### Install Arvados Root Certificate on Debian/Ubuntu (Bash)

Source: https://doc.arvados.org/v3.1/install/arvbox

Steps to install the Arvados root certificate into the system's trusted certificate store on Debian/Ubuntu-based systems. This ensures command-line tools can access Arvados securely. Requires the certificate file to have a .crt extension.

```bash
cp arvbox-root-cert.crt /usr/local/share/ca-certificates/arvados-snakeoil-ca.crt
/usr/sbin/update-ca-certificates
```

--------------------------------

### Reconfigure Arvados API Server

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Re-runs the post-install script for arvados-api-server to set up the database. This is typically used after correcting database connection information.

```bash
dpkg-reconfigure arvados-api-server
```

--------------------------------

### Manage Keepproxy Service with systemd

Source: https://doc.arvados.org/v3.1/install/install-keepproxy

Enables and starts the keepproxy service using systemd, and checks its status. It also provides instructions on how to view logs for troubleshooting if the service is not running.

```bash
# systemctl enable --now keepproxy
# systemctl status keepproxy
[...] # Check logs for errors:
# journalctl --since -5min -u keepproxy

```

--------------------------------

### Enable and Start Docker Service

Source: https://doc.arvados.org/v3.1/install/install-docker

Ensures the Docker service is enabled to start on boot and starts it immediately. Requires root privileges.

```bash
# systemctl enable --now docker
```

--------------------------------

### Arvados-in-a-box Quick Start (Bash)

Source: https://doc.arvados.org/v3.1/install/arvbox

Provides commands to download, make executable, and start the Arvados-in-a-box development environment. It also includes adding a user to the environment. Requires Linux and Docker.

```bash
curl -O https://git.arvados.org/arvados.git/blob_plain/refs/heads/3.1-release:/tools/arvbox/bin/arvbox
chmod +x arvbox
./arvbox start localdemo

./arvbox adduser demouser demo@example.com
```

--------------------------------

### Initialize Arvados Installer without Terraform

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Initializes the Arvados installer script for a multi-host setup on AWS without using Terraform. This is an alternative for environments where Terraform is not used for infrastructure provisioning.

```bash
CLUSTER=xarv1
./installer.sh initialize ~/setup-arvados-${CLUSTER} multiple_hosts multi_host/aws
cd ~/setup-arvados-${CLUSTER}
```

--------------------------------

### Clone Arvados Documentation Repository

Source: https://doc.arvados.org/v3.1/user/cwl/cwl-runner

Fetches the Arvados source code repository, which contains documentation and tutorial files, including those for CWL workflows. This is the first step to obtain the necessary files for the tutorial.

```bash
~$ git clone https://git.arvados.org/arvados.git
~$ cd arvados/doc/user/cwl/bwa-mem
```

--------------------------------

### Example Boot Probe Commands for Cloud VM Instances

Source: https://doc.arvados.org/v3.1/architecture/dispatchcloud

Demonstrates commands used to probe the readiness of a cloud VM instance before scheduling containers. It includes checking systemd status or a custom boot completion file.

```shell
systemctl is-system-running
```

```shell
cat /var/run/boot.complete
```

--------------------------------

### Install and Initialize Arvados R SDK

Source: https://doc.arvados.org/v3.1/sdk/R/index

Installs the ArvadosR package from a repository and loads the library. It also demonstrates initializing the Arvados API client, optionally with a token, host, and retry count.

```R
install.packages("ArvadosR", repos=c("https://r.arvados.org", getOption("repos")["CRAN"]))
library('ArvadosR')
```

```R
# use environment variables ARVADOS_API_TOKEN and ARVADOS_API_HOST
arv <- Arvados$new()

# provide them explicitly
arv <- Arvados$new("your Arvados token", "example.arvadosapi.com")
```

```R
arv <- Arvados$new("your Arvados token", "example.arvadosapi.com", numRetries = 3)
```

```R
arv$setNumRetries(5)
```

--------------------------------

### Generate PostgreSQL Password

Source: https://doc.arvados.org/v3.1/install/install-api-server

Generates a random 25-character password for the PostgreSQL user. This password is used during database setup and for configuring the Arvados Rails API.

```shell
# su postgres
postgres$ tr -dc 0-9a-zA-Z </dev/urandom | head -c25; echo
yourgeneratedpassword

```

--------------------------------

### Initialize Arvados Go SDK Client

Source: https://doc.arvados.org/v3.1/sdk/go/example

Demonstrates how to initialize the Arvados client in Go, which is necessary for making API calls. It handles potential errors during client setup.

```go
import (
  "git.arvados.org/arvados.git/sdk/go/arvados"
  "git.arvados.org/arvados.git/sdk/go/arvadosclient"
)

func main() {
  arv, err := arvadosclient.MakeArvadosClient()
  if err != nil {
    log.Fatalf("Error setting up arvados client %s", err.Error())
  }
}
```

--------------------------------

### Go: Arvados API Client Setup and User Information Retrieval

Source: https://doc.arvados.org/v3.1/sdk/go/example

Demonstrates how to initialize the Arvados API client using the `arvadosclient` package and retrieve the current user's full name and UUID. It includes error handling for client setup and API calls. Dependencies include the 'fmt', 'log', and 'arvadosclient' packages.

```go
// Copyright (C) The Arvados Authors. All rights reserved.
// 
// SPDX-License-Identifier: CC-BY-SA-3.0

package main


// *******************
// Import the modules.
//
// Our examples don't use keepclient, but they do use fmt and log to
// display output.

import (
        "fmt"
        "git.arvados.org/arvados.git/sdk/go/arvadosclient"
        "log"
)

func main() {


        // ********************************
        // Set up an API client user agent.
        //

        arv, err := arvadosclient.MakeArvadosClient()
        if err != nil {
                log.Fatalf("Error setting up arvados client %s", err.Error())
        }


        // *****************************************
        // Print the full name of the current user.
        //

        type user struct {
                // Remember to start each field name with a capital letter,
                // otherwise it won't get populated by the arvados client because
                // the field will be invisible to it.
                Uuid     string `json:"uuid"`
                FullName string `json:"full_name"`
        }

        var u user
        err = arv.Call("GET", "users", "", "current", nil, &u)

        if err != nil {
                log.Fatalf("error querying current user", err.Error())
        }

        log.Printf("Logged in as %s (uuid %s)", u.FullName, u.Uuid)


        // ********************************************************
        // Print all fields from the first five collections returned.
        //
        // Note that some fields, are not returned by default and have to be
        // requested. See below for an example.

        var results map[string]interface{}

        params := arvadosclient.Dict{"limit": 5}

        err = arv.List("collections", params, &results)
        if err != nil {
                log.Fatalf("error querying collections", err.Error())
        }

        printArvadosResults(results)


        // *********************************************************
        // Print some fields from the first two collections returned.
        //
        // We also print manifest_test, which has to be explicitly requested.
        //

        collection_fields_wanted := []string{"manifest_text", "owner_uuid", "uuid"}
        params = arvadosclient.Dict{"limit": 2, "select": collection_fields_wanted}

        err = arv.List("collections", params, &results)
        if err != nil {
                log.Fatalf("error querying collections", err.Error())
        }

        printArvadosResults(results)
}


// A helper method which will print out a result map returned by
// arvadosclient.
func printArvadosResults(results map[string]interface{}) {
        for key, value := range results {
                // "items", if it exists, holds a map.
                // So we print it prettily below.
                if key != "items" {
                        fmt.Println(key, ":", value)
                }
        }

        if value, ok := results["items"]; ok {
                items := value.([]interface{})
                for index, item := range items {
                        fmt.Println("===========  ", index, "  ===========")
                        item_map := item.(map[string]interface{})
                        if len(item_map) == 0 {
                                fmt.Println("item", index, ": empty map")
                        } else {
                                for k, v := range item_map {
                                        fmt.Println(index, k, ":", v)
                                }
                        }
                }
        }
}

```

--------------------------------

### Arvados Dispatcher SSH Private Key

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Provides an example of an SSH private key used by `arvados-dispatch-cloud` to connect to compute nodes. The key should be generated following specific documentation and stored securely.

```shell
DISPATCHER_SSH_PRIVKEY="-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
s4VY40kNxs6MsAAAAPbHVjYXNAaW5zdGFsbGVyAQIDBA==
-----END OPENSSH PRIVATE KEY-----"

```

--------------------------------

### Terraform Configuration for Arvados Data Storage

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Example Terraform tfvars file for configuring Arvados data storage, including options for using an external database. This file allows for customization of stateful components.

```hcl
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: CC-BY-SA-3.0

# Set to true if the database server won't be running in any service instance.
# Default: false
# use_external_db = true
```

--------------------------------

### Copy Tutorial Data using arv-copy

Source: https://doc.arvados.org/v3.1/user/cwl/cwl-runner

Uses the `arv-copy` command to transfer tutorial data from a remote Arvados instance ('pirca') to your local Arvados instance. This is essential for running the workflow examples if you don't want to download and upload files manually.

```bash
~$ arv-copy --src pirca --dst settings 2463fa9efeb75e099685528b3b9071e0+438
~$ arv-copy --src pirca --dst settings ae480c5099b81e17267b7445e35b4bc7+180
```

--------------------------------

### Setup User

Source: https://doc.arvados.org/v3.1/sdk/R/arvados/Arvados

Performs a full setup for a user, including potentially creating a virtual machine login and sending a notification email.

```APIDOC
## POST /users/{uuid}/setup

### Description
Convenience method to "fully" set up a user record with a virtual machine login and notification email.

### Method
POST

### Endpoint
/users/{uuid}/setup

### Parameters
#### Path Parameters
- **uuid** (string) - Optional - UUID of an existing user record to set up.

#### Request Body
- **user** (object) - Optional - Attributes of a new user record to set up.
- **repoName** (string) - Optional - This parameter is obsolete and ignored.
- **vmUUID** (string) - Optional - If given, setup creates a login link to allow this user to access the Arvados virtual machine with this UUID.
- **sendNotificationEmail** (boolean) - Optional - If true, send an email to the user notifying them they can now access this Arvados cluster.

### Request Example
```json
{
  "vmUUID": "vm-12345",
  "sendNotificationEmail": true
}
```

### Response
#### Success Response (200)
- **user** (object) - The set up User object.

#### Response Example
```json
{
  "uuid": "123e4567-e89b-12d3-a456-426614174000",
  "name": "user1",
  "login_link": "https://arvados.example.com/login/vm-12345"
}
```
```

--------------------------------

### Terraform Configuration for Arvados VPC

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Example Terraform tfvars file for configuring Arvados VPC, including AWS region, cluster name, domain name, and optional networking settings. This file is crucial for setting up the network infrastructure.

```hcl
# Copyright (C) The Arvados Authors. All rights reserved.
#
# SPDX-License-Identifier: CC-BY-SA-3.0

# Main cluster configurations. No sensible defaults provided for these:
# region_name = "us-east-1"
# cluster_name = "xarv1"
# domain_name = "xarv1.example.com"

# Uncomment this to create an non-publicly accessible Arvados cluster
# private_only = true

# Optional networking options. Set existing resources to be used instead of
# creating new ones.
# NOTE: We only support fully managed or fully custom networking, not a mix of both.
#
# vpc_id = "vpc-aaaa"
# sg_id = "sg-bbbb"
# public_subnet_id = "subnet-cccc"
# private_subnet_id = "subnet-dddd"
#
# RDS related parameters:
# use_rds = true
# additional_rds_subnet_id = "subnet-eeee"

# Optional custom tags to add to every resource. Default: {}
# custom_tags = {
#   environment = "production"
#   project = "Phoenix"
#   owner = "jdoe"
# }

# Optional cluster service nodes configuration:
#
# List of node names which either will be hosting user-facing or internal
# services. Defaults:
# user_facing_hosts = [ "controller", "workbench" ]
# internal_service_hosts = [ "keep0", "shell" ]
#
# Map assigning each node name an internal IP address. Defaults:
# private_ip = {
#   controller = "10.1.1.11"
#   workbench = "10.1.1.15"
#   shell = "10.1.2.17"
#   keep0 = "10.1.2.13"
# }
#
# Map assigning DNS aliases for service node names. Defaults:
# dns_aliases = {
#   workbench = [
#     "ws",
#     "workbench2",
#     "webshell",
#     "keep",
#     "download",
#     "prometheus",
#     "grafana",
#     "loki",
#     "*.collections"
#   ]
# }
```

--------------------------------

### Deploy Arvados

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Executes the Arvados installation script in deploy mode to conduct the full installation process.

```bash
./installer.sh deploy

```

--------------------------------

### Workbench Configuration for Getting Started Popup

Source: https://doc.arvados.org/v3.1/admin/upgrading

Configuration setting for Workbench to enable or disable the 'Getting Started' popup for new users. Setting `enable_getting_started_popup` to `true` will display the popup.

```yaml
enable_getting_started_popup: true
```

--------------------------------

### Install keepstore on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/install/install-keepstore

Installs the keepstore package using the APT package manager on Debian-based systems.

```bash
# apt install keepstore

```

--------------------------------

### Configure Docker Daemon Default Ulimits

Source: https://doc.arvados.org/v3.1/install/install-docker

Example of setting a default ulimit for the Docker daemon to allow containers to open a higher number of files. This is a configuration option for the Docker daemon.

```bash
--default-ulimit nofile=10000:10000
```

--------------------------------

### R SDK Installation

Source: https://doc.arvados.org/v3.1/sdk/R/index

Instructions for installing the Arvados R SDK, including R version requirements and necessary system packages.

```APIDOC
## R SDK Installation

Minimum R version required to run ArvadosR is 3.3.0.

```R
install.packages("ArvadosR", repos=c("https://r.arvados.org", getOption("repos")["CRAN"]))
library('ArvadosR')
```

**Note** On Linux, you may have to install supporting packages.
On Red Hat, AlmaLinux, and Rocky Linux, this is: ` yum install libxml2-devel openssl-devel curl-devel `
On Debian and Ubuntu, this is: ` apt-get install build-essential libxml2-dev libssl-dev libcurl4-gnutls-dev `
```

--------------------------------

### Arvbox Command-Line Interface

Source: https://doc.arvados.org/v3.1/install/arvbox

This section lists the available commands for the 'arvbox' tool, which manages Arvados development environments within Docker containers. It covers starting, stopping, status checks, shell access, database operations, logging, file management, and user administration.

```shell
$ arvbox
Arvados-in-a-box             https://doc.arvados.org/install/arvbox.html

start|run <config> [tag]   start arvbox container
stop               stop arvbox container
restart <config>   stop, then run again
status             print some information about current arvbox
ip                 print arvbox docker container ip address
host               print arvbox published host
shell              enter shell as root
ashell             enter shell as 'arvbox'
psql               enter postgres console
open               open arvbox workbench in a web browser
root-cert          get copy of root certificate
update  <config>   stop, pull latest image, run
build   <config>   build arvbox Docker image
reboot  <config>   stop, build arvbox Docker image, run
rebuild <config>   build arvbox Docker image, no layer cache
checkpoint         create database backup
restore            restore checkpoint
hotreset           reset database and restart API without restarting container
reset              delete arvbox arvados data (be careful!)
destroy            delete all arvbox code and data (be careful!)
log <service>      tail log of specified service
ls <options>       list directories inside arvbox
cat <files>        get contents of files inside arvdata
pipe               run a bash script piped in from stdin
sv <start|stop|restart> <service>
                   change state of service inside arvbox
clone <from> <to>  clone dev arvbox
adduser <username> <email> [password]
                   add a user login
removeuser <username>
                   remove user login
listusers          list user logins

```

--------------------------------

### Initialize Arvados Installer with Terraform

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Initializes the Arvados installer script to set up a cluster with multiple hosts using Terraform on AWS. This command assumes Terraform is installed and configured.

```bash
CLUSTER=xarv1
./installer.sh initialize ~/setup-arvados-${CLUSTER} multiple_hosts multi_host/aws terraform/aws
cd ~/setup-setup-arvados-${CLUSTER}
```

--------------------------------

### Install keepstore on Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/install/install-keepstore

Installs the keepstore package using the DNF package manager on Red Hat-based systems.

```bash
# dnf install keepstore

```

--------------------------------

### Install arvados-workbench2 on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/install/install-workbench2-app

Install the Arvados Workbench 2 package using the `apt` package manager on Debian-based systems.

```bash
# apt install arvados-workbench2

```

--------------------------------

### Create Arvados Collection (Go)

Source: https://doc.arvados.org/v3.1/sdk/go/example

Provides an example of creating a new Arvados collection using the Go SDK. It shows how to specify collection properties like its name.

```go
var collection arvados.Collection
err := api.Create("collections", Dict{"collection": Dict{"name": "create example"}}, &collection)
```

--------------------------------

### Command Line SDK Usage Examples

Source: https://doc.arvados.org/v3.1/sdk/index

Provides examples of using the Arvados command-line interface (CLI) tool, 'arv'. This SDK is essential for scripting and direct interaction with Arvados.

```bash
# Example: Uploading a file using arv-put
arv put --path /path/to/local/file
```

```bash
# Example: Downloading a file using arv-get
arv get --uuid some-uuid
```

--------------------------------

### arvados-server cloudtest help

Source: https://doc.arvados.org/v3.1/admin/cloudtest

Displays the help message for the `arvados-server cloudtest` command, listing all available options and their descriptions.

```bash
$ arvados-server cloudtest -help
```

--------------------------------

### Install arvados-workbench2 on Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/install/install-workbench2-app

Install the Arvados Workbench 2 package using the `dnf` package manager on Red Hat-based systems.

```bash
# dnf install arvados-workbench2

```

--------------------------------

### Ruby SDK Examples

Source: https://doc.arvados.org/v3.1/sdk/index

Illustrates basic usage patterns for the Arvados Ruby SDK. Similar to Go, Java, and R, this SDK might not cover all Arvados functionalities.

```ruby
# Example: Requiring the Arvados Ruby gem
require 'arvados'

# ... use arvados methods
```

--------------------------------

### Run Arvados Terraform Installer

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Executes the Arvados installer script to provision the AWS infrastructure. The script utilizes Terraform and outputs essential cluster information upon successful completion.

```bash
./installer.sh terraform
```

--------------------------------

### Go SDK Examples

Source: https://doc.arvados.org/v3.1/sdk/index

Illustrates how to use the Arvados Go SDK for interacting with the Arvados API. Note that this SDK may have fewer features compared to the Python SDK.

```go
// Example: Initializing the Arvados Go client
import (
	"arvados"
)

client, err := arvados.NewClient()
if err != nil {
	panic(err)
}
```

--------------------------------

### Configure OpenID Connect Login for Arvados

Source: https://doc.arvados.org/v3.1/install/setup-login

This snippet demonstrates configuring Arvados to use OpenID Connect for user authentication with third-party providers like GitHub or Auth0. It involves registering Arvados with the provider to get an issuer URL, Client ID, and Client Secret, which are then added to the Arvados config.yml.

```yaml
Login:
  OpenIDConnect:
    Enable: true
    Issuer: https://accounts.example.com/
    ClientID: "0123456789abcdef"
    ClientSecret: "zzzzzzzzzzzzzzzzzzzzzzzz"
```

--------------------------------

### Clone Arvados Tutorial Repository and Navigate

Source: https://doc.arvados.org/v3.1/user/tutorials/wgs-tutorial

This command clones the Arvados tutorial repository from GitHub and then changes the current directory to the WGS tutorial folder, which contains the necessary workflow files.

```bash
$ git clone https://github.com/arvados/arvados-tutorial.git
$ cd arvados-tutorial/WGS-processing
```

--------------------------------

### Go SDK Usage Example

Source: https://doc.arvados.org/v3.1/sdk

Provides examples for utilizing the Go SDK to interact with the Arvados API. While the Go SDK is available, it may have fewer features or less comprehensive documentation compared to the Python SDK.

```go
package main

import (
	"fmt"
	"github.com/curoverse/go-arvados/sdk/go/arvados"
)

func main() {
	arvadosClient := arvados.NewClient("") // Use default config

	// Example: List projects (simplified)
	projects, err := arvadosClient.Projects.List(nil)
	if err != nil {
		fmt.Printf("Error listing projects: %v\n", err)
		return
	}
	for _, project := range projects {
		fmt.Println(project.UUID)
	}
}

```

--------------------------------

### Install crunch-dispatch-slurm package

Source: https://doc.arvados.org/v3.1/install/crunch2-slurm/install-dispatch

Installs the crunch-dispatch-slurm package using apt. This is the initial step to set up the service.

```bash
# apt install crunch-dispatch-slurm
```

--------------------------------

### Install Arvados CA Certificate on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Copies the generated Arvados CA certificate to the system's CA certificates directory and updates the certificate store.

```bash
cp xarv1.example.com-arvados-snakeoil-ca.crt /usr/local/share/ca-certificates/arvados-snakeoil-ca.crt
/usr/sbin/update-ca-certificates

```

--------------------------------

### Install Go SDK

Source: https://doc.arvados.org/v3.1/sdk/go/index

Installs the Go (Golang) SDK for Arvados, enabling easy API calls. It fetches code and dependencies using go tools.

```go
import (
        "git.arvados.org/arvados.git/sdk/go/arvadosclient"
        "git.arvados.org/arvados.git/sdk/go/keepclient"
)
```

--------------------------------

### Configure shellinabox on Debian/Ubuntu

Source: https://doc.arvados.org/v3.1/install/install-webshell

Configures the shellinabox service by setting the listening port and service arguments in `/etc/default/shellinabox`. It then enables and starts the shellinabox service using systemctl.

```bash
# TCP port that shellinboxd's webserver listens on
SHELLINABOX_PORT=4200

# SSL is disabled because it is terminated in Nginx. Adjust as needed.
SHELLINABOX_ARGS="--disable-ssl --no-beep --service=/shell.ClusterID.example.com:AUTH:HOME:SHELL"

```

```bash
# systemctl enable shellinabox
# systemctl start shellinabox

```

--------------------------------

### Grant Passwordless Sudo Access

Source: https://doc.arvados.org/v3.1/install/salt-single-host

This snippet shows how to configure `/etc/sudoers.d/arvados_passwordless` to grant members of the 'sudo' group passwordless access to execute any command. This is a prerequisite for the Arvados installation.

```shell
%sudo ALL=(ALL:ALL) NOPASSWD:ALL
```

--------------------------------

### Arvados Client Initialization and Collection Management

Source: https://doc.arvados.org/v3.1/sdk/R/arvados/Arvados

Demonstrates initializing an Arvados client with a token and hostname, and performing operations on collections like getting, listing, deleting, updating, and creating.

```R
## Not run: 
arv <- Arvados$new("your Arvados token", "example.arvadosapi.com")

collection <- arv$collections.get("uuid")

collectionList <- arv$collections.list(list(list("name", "like", "Test%")))
collectionList <- listAll(arv$collections.list, list(list("name", "like", "Test%")))

deletedCollection <- arv$collections.delete("uuid")

updatedCollection <- arv$collections.update(list(name = "New name", description = "New description"),
                                            "uuid")

createdCollection <- arv$collections.create(list(name = "Example",
                                                 description = "This is a test collection"))

## End(Not run)
```

--------------------------------

### Export AWS Credentials

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Sets the AWS access key ID and secret access key as environment variables, which are required for the Arvados installer script to authenticate with AWS services.

```bash
export AWS_ACCESS_KEY_ID="anaccesskey"
export AWS_SECRET_ACCESS_KEY="asecretkey"
```

--------------------------------

### Arvados users_setup() method

Source: https://doc.arvados.org/v3.1/sdk/R/arvados/Arvados

Sets up a user record with a virtual machine login and notification email. It can set up an existing user by UUID or create a new user with provided attributes. Optionally, a virtual machine UUID can be provided for login access and a flag to send a notification email. Returns the User object.

```R
Arvados$users_setup(
  uuid = NULL,
  user = NULL,
  repoName = NULL,
  vmUUID = NULL,
  sendNotificationEmail = NULL
)
```

--------------------------------

### Install Arvados CA Certificate on Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/install/salt-single-host

Copies the generated Arvados CA certificate to the system's trust anchor directory and updates the certificate store.

```bash
cp xarv1.example.com-arvados-snakeoil-ca.crt /etc/pki/ca-trust/source/anchors/
/usr/bin/update-ca-trust

```

--------------------------------

### Install Arvados API Server and Controller on Red Hat/AlmaLinux/Rocky Linux

Source: https://doc.arvados.org/v3.1/install/install-api-server

Installs the Arvados API server and controller packages using the DNF package manager on Red Hat Enterprise Linux, AlmaLinux, or Rocky Linux. Ensure the 'devel' repository is enabled for your distribution version.

```bash
# dnf install arvados-api-server arvados-controller
```

--------------------------------

### Define GET access for collections with trailing slash

Source: https://doc.arvados.org/v3.1/admin/scoped-tokens

This scope permits GET or HEAD requests for paths starting with '/arvados/v1/collections/'. Requests to list collections directly will be denied if they don't end with a slash.

```JSON
["GET", "/arvados/v1/collections/"]
```

--------------------------------

### Initialize Arvados SDK (Ruby)

Source: https://doc.arvados.org/v3.1/sdk/ruby/example

Initializes the Arvados SDK by importing the module and setting up an API client user agent. The SDK dynamically retrieves API methods based on the server version.

```ruby
require 'arvados'
arv = Arvados.new(apiVersion: 'v1')
```

--------------------------------

### Run Arvados Diagnostics

Source: https://doc.arvados.org/v3.1/install/salt-multi-host

Executes diagnostic checks for the Arvados cluster. Requires either an internal or external client connection depending on the execution environment. Ensure 'arvados-client' and 'docker' are installed if running from a local system.

```shell
./installer.sh diagnostics (-internal-client|-external-client)
```

--------------------------------

### Gradle Configuration for Arvados Java SDK

Source: https://doc.arvados.org/v3.1/sdk/java-v2/index

Example build.gradle file demonstrating how to include the Arvados Java SDK v2 using Maven Central. This sets up the necessary repositories and dependencies for using the SDK in a Java project.

```gradle
apply plugin: 'application'
apply plugin: 'java-library'
apply plugin: 'maven'

repositories {
    mavenCentral()
}

application {
    mainClassName = "org.arvados.example.CollectionExample"
}

dependencies {
    api 'org.arvados:arvados-java-sdk:0.1.1'
}

```

--------------------------------

### R SDK Examples

Source: https://doc.arvados.org/v3.1/sdk/index

Shows examples of using the Arvados R SDK for data analysis and interaction with Arvados. This SDK's documentation and feature set may be limited.

```r
# Example: Loading the Arvados R SDK
library(arvados)

# ... use arvados functions
```