### Verify Docker Installation

Source: https://openproblems.bio/documentation/fundamentals/requirements

Runs a simple 'hello-world' container to confirm Docker is installed and functioning correctly.

```bash
docker run hello-world
```

--------------------------------

### Run Viash Component

Source: https://openproblems.bio/documentation/create_task/getting_started

Executes a specific Viash component. Arguments after '--' are passed directly to the component. This example runs the logistic regression component with specified training and testing data.

```shell
viash run src/methods/logistic_regression/config.vsh.yaml -- \
  --input_train resources_test/task_template/cxg_mouse_pancreas_atlas/train.h5ad \
  --input_test resources_test/task_template/cxg_mouse_pancreas_atlas/test.h5ad \
  --output output.h5ad
```

--------------------------------

### Install Nextflow

Source: https://openproblems.bio/documentation/fundamentals/requirements

Installs Nextflow, a workflow management system. It downloads the Nextflow binary and places it in the user's local bin directory.

```bash
curl -s https://get.nextflow.io | bash
mkdir -p "$HOME/.local/bin"
mv nextflow "$HOME/.local/bin"
```

--------------------------------

### Clone Open Problems Repository with Git

Source: https://openproblems.bio/documentation/create_component/getting_started

Command to clone the forked Open Problems repository and its submodules recursively. Ensures all necessary components are downloaded for local development.

```bash
git clone --recursive <forked repository URL>
cd <repository name>
```

--------------------------------

### Install Viash

Source: https://openproblems.bio/documentation/fundamentals/requirements

Installs Viash, a command-line tool. It downloads the Viash binary and moves it to a directory in the user's PATH.

```bash
curl -fsSL dl.viash.io | bash
mkdir -p "$HOME/.local/bin"
mv viash "$HOME/.local/bin"
```

--------------------------------

### Test Single Viash Component with Git

Source: https://openproblems.bio/documentation/create_component/getting_started

Command to run predefined unit tests for a specific Viash component configuration file. This helps verify the correctness of individual methods.

```bash
viash test src/methods/logistic_regression/config.vsh.yaml
```

--------------------------------

### Verify Nextflow and Docker Integration

Source: https://openproblems.bio/documentation/fundamentals/requirements

Runs a sample Nextflow pipeline ('hello') using Docker, verifying both Nextflow and Docker are functional and integrated.

```bash
nextflow run hello -with-docker
```

--------------------------------

### Test All Viash Components in Namespace

Source: https://openproblems.bio/documentation/create_component/getting_started

Command to execute unit tests for all components within a namespace, utilizing parallel processing and Docker for execution. Provides a comprehensive test overview.

```bash
viash ns test --parallel --engine docker
```

--------------------------------

### Install Docker on Debian/Ubuntu

Source: https://openproblems.bio/documentation/fundamentals/requirements

Installs Docker and its service on Debian/Ubuntu systems. Enables Docker to run and adds the current user to the docker group.

```bash
sudo apt update
sudo apt install docker.io
sudo systemctl enable docker
sudo usermod -aG docker $USER
```

--------------------------------

### Download Test Resources with Shell Script

Source: https://openproblems.bio/documentation/create_component/getting_started

Executes a shell script to download necessary test resources for validating existing components and developing new unit tests within the Open Problems project.

```bash
scripts/sync_resources.sh
```

--------------------------------

### Clone and Initialize Task Repository - Git

Source: https://openproblems.bio/documentation/create_task/getting_started

Clones a forked task repository from a remote URL and initializes its submodules. This is essential for obtaining the task's codebase and any associated data or utilities.

```bash
git clone --recursive <forked repository URL>
cd <repository name>

# If submodules are not updated automatically:
git submodule update --init --recursive
```

--------------------------------

### Verify Java Version

Source: https://openproblems.bio/documentation/fundamentals/requirements

Command to verify the installed Java version. This ensures that Java 11 or a later version is available.

```bash
java --version
```

--------------------------------

### Install Java Development Kit (OpenJDK 11) on Debian/Ubuntu

Source: https://openproblems.bio/documentation/fundamentals/requirements

Installs the OpenJDK 11 Development Kit on Debian/Ubuntu systems, required for Java-based development. Includes verification step.

```bash
sudo apt-get update
sudo apt-get install openjdk-11-jdk
java -version
```

--------------------------------

### Update Git Submodules for Open Problems

Source: https://openproblems.bio/documentation/create_component/getting_started

Command to initialize and update git submodules, typically used if the 'common' submodule is not visible after initial cloning.

```bash
git submodule update --init --recursive
```

--------------------------------

### Verify Viash Version

Source: https://openproblems.bio/documentation/fundamentals/requirements

Checks the installed version of Viash to ensure it meets the minimum requirement.

```bash
viash -v
```

--------------------------------

### Run Viash Component for Logistic Regression

Source: https://openproblems.bio/documentation/create_component/getting_started

Command to execute a specific Viash component (logistic_regression) with provided input training and testing data, outputting the results. Arguments after '--' are component-specific.

```bash
viash run src/methods/logistic_regression/config.vsh.yaml -- \
 --input_train resources_test/task_template/cxg_mouse_pancreas_atlas/train.h5ad \
 --input_test resources_test/task_template/cxg_mouse_pancreas_atlas/test.h5ad \
 --output output.h5ad
```

--------------------------------

### Verify Docker Installation

Source: https://openproblems.bio/documentation/more_information/troubleshooting

These commands verify the Docker installation and configuration. 'groups' shows the user's current group memberships, and 'docker run hello-world' tests the Docker daemon's functionality by pulling and running a test image.

```bash
groups
```

```bash
docker run hello-world
```

--------------------------------

### Install Bash on Debian/Ubuntu

Source: https://openproblems.bio/documentation/fundamentals/requirements

Installs the Bash shell on Debian/Ubuntu based systems. This is a fundamental requirement for command-line operations.

```bash
sudo apt-get update
sudo apt-get install bash
```

--------------------------------

### Configure Custom Docker Image Setup with Python Dependencies

Source: https://openproblems.bio/documentation/create_component/add_a_method

Defines the minimum setup for a custom Docker image, including apt and Python packages. It specifies required Python libraries like anndata, scanpy, pyyaml, requests, and jsonschema, and links to a GitHub repository for additional Open Problems Python packages.

```yaml
platforms:
  - type: docker
    Image: your custom image
    setup:
      - type: apt
        packages:
          - procps
      - type: python
        packages:
          - anndata~=0.10.0
          - scanpy~=1.10.0
          - pyyaml
          - requests
          - jsonschema
        github:
          - "openproblems-bio/core#subdirectory=packages/python/openproblems"

```

--------------------------------

### Enable Docker Service on Boot

Source: https://openproblems.bio/documentation/more_information/troubleshooting

This command enables the Docker service to start automatically on system boot. It is a common step to resolve issues where the Docker daemon is not running. It's often followed by a command to test the Docker installation.

```bash
sudo systemctl enable docker
```

--------------------------------

### Run Component Locally (Shell)

Source: https://openproblems.bio/documentation/create_component/add_a_control

Example command to run a component on local files using 'viash run'. This allows for testing the component with specific input and output files before deployment.

```shell
viash run src/tasks/label_projection/control_methods/my_python_method/config.vsh.yaml -- \
  --input_train resources_test/task_template/cxg_mouse_pancreas_atlas/train.h5ad \
  --input_test resources_test/task_template/cxg_mouse_pancreas_atlas/test.h5ad \
  --input_solution resources_test/task_template/cxg_mouse_pancreas_atlas/solution.h5ad \
  --output output.h5ad

```

--------------------------------

### List Java Versions in Linux

Source: https://openproblems.bio/documentation/more_information/troubleshooting

This command lists all installed Java versions found in the /usr/lib/jvm/ directory. It helps in identifying available Java installations on a Linux system, which is useful for managing different Java environments.

```bash
ls -1 /usr/lib/jvm/
```

--------------------------------

### Example Python Method Configuration

Source: https://openproblems.bio/documentation/create_component/add_a_method

A generated Viash configuration file for a Python method. This YAML file defines metadata such as the component's name, label, summary, and a detailed description, along with input/output specifications and parameters.

```yaml
#| eval: false
# Viash component configuration file for a Python method.

# Component metadata
info:
  # API specification for input/output files, common parameters, and unit tests.
  merge: |
    
  # Unique identifier for the method (snakecase).
  name: my_python_method
  # Human-readable, short label for summary tables and visualizations.
  label: My Python Method
  # One-sentence summary of the method's purpose and methodology.
  summary: A method for analyzing omics data using Python.
  # Longer description for reference documentation.
  description: |
    This method performs a specific analysis on omics data using Python.
    It is designed to be integrated into the Open Problems framework.

# Specifies the script used to run the method.
executable: script.py

# Defines the required Docker image for running the method.
docker_image: openproblems/base_python:latest

# Specifies the required R packages for the method.
r_packages: []

# Specifies the required Python packages for the method.
python_packages: []

# Defines the input and output parameters for the method.
parameters:
  - name: "input_files"
    type: "file"
    direction: "input"
    description: "Input data files."
  - name: "output_files"
    type: "file"
    direction: "output"
    description: "Output data files."

# Unit test for the method.
test:
  
  # Input data for the unit test.
  input:
    
  # Expected output for the unit test.
  expected_output:
    
```

--------------------------------

### Build Docker Containers with Viash

Source: https://openproblems.bio/documentation/fundamentals/common_commands

Builds all Docker containers for the project's components. It supports parallel builds and utilizes cached builds for efficiency with the '--setup cachedbuild' option.

```shell
viash ns build --parallel --setup cachedbuild
```

--------------------------------

### Example Git Branch Naming for Tests/Experiments

Source: https://openproblems.bio/documentation/reference/naming

Provides an example of naming a git branch for experimental purposes or refactoring outside of a formal issue, using the 'test' category and 'no-ref'.

```shell
git branch test/no-ref/refactor-components-with-atomic-design
```

--------------------------------

### Viash Config File for Control Method

Source: https://openproblems.bio/documentation/create_component/add_a_control

Example of a Viash configuration file generated for a control method. This file contains metadata such as the component's name, label, summary, and description, as well as specifications for input/output files and common parameters.

```yaml
info:
  "#merge": |
    input_from:
      - "task_schema.yaml"
  name: "my_python_method"
  label: "My Python Method"
  summary: "A brief summary of the python method."
  description: |
    A more detailed description of the python method.

    Can span multiple lines.

    It is recommended to include information about:
    * The methodology
    * The underlying algorithm
    * Any assumptions made
    * Potential limitations
    * Key parameters

    etc.

parameters:
  "#merge":
    - "task_parameters.yaml"

resources:
  "#merge":
    - "task_resources.yaml"

test:
  "#merge":
    - "task_test.yaml"

```

--------------------------------

### Run Open Problems Bio Component Locally

Source: https://openproblems.bio/documentation/create_component/add_a_method

Demonstrates how to run a component locally using the `viash run` command. It specifies the component's configuration file and provides example input and output file paths for testing the component's execution with specific data.

```bash
viash run src/methods/my_python_method/config.vsh.yaml -- \
  --input_train resources_test/task_template/cxg_mouse_pancreas_atlas/train.h5ad \
  --input_test resources_test/task_template/cxg_mouse_pancreas_atlas/test.h5ad \
  --output output.h5ad

```

--------------------------------

### Rebuild Docker Container with Cached Build

Source: https://openproblems.bio/documentation/create_component/add_a_method

Command to rebuild the Docker container after modifying component dependencies. It uses `viash run` with a configuration file and the `--setup cachedbuild` argument to ensure the build process is efficient.

```bash
#| echo: false
viash run src/methods/my_python_method/config.vsh.yaml -- \
  ---setup cachedbuild

```

--------------------------------

### Python Metric Script Structure Example

Source: https://openproblems.bio/documentation/create_component/add_a_metric

This is a placeholder for the Python script that implements the metric's logic. It typically includes sections for imports, argument parsing, reading input data, generating results, and writing output.

```python
cat src/metrics/my_python_metric/script.py
```

--------------------------------

### Add Parameters to Viash Config

Source: https://openproblems.bio/documentation/advanced_topics/create_a_dataset_loader

Example of adding custom arguments (`--n_obs`, `--n_vars`) to the `functionality.arguments` section in a Viash configuration file. These parameters allow users to control the generation of the dataset.

```yaml
arguments:
  - name: "--n_obs"
    type: "integer"
    description: "Number of cells to generate."
    default: 100
  - name: "--n_vars"
    type: "integer"
    description: "Number of genes to generate."
    default: 100
```

--------------------------------

### Run Method using Viash

Source: https://openproblems.bio/documentation/create_task/dataset_processor

This command executes a specific method (e.g., k-nearest neighbors) using the Viash tool. It requires Viash to be installed and a configuration file for the method. The script takes input training and testing data and generates a prediction file.

```bash
viash run src/methods/knn/config.vsh.yaml -- \
    --input_train $DATASET_DIR/cxg_mouse_pancreas_atlas/train.h5ad \
    --input_test $DATASET_DIR/cxg_mouse_pancreas_atlas/test.h5ad \
    --output $DATASET_DIR/cxg_mouse_pancreas_atlas/prediction.h5ad
```

--------------------------------

### Set JAVA_HOME Environment Variable

Source: https://openproblems.bio/documentation/more_information/troubleshooting

This command sets the JAVA_HOME environment variable to a specific Java installation path. It is crucial for applications that rely on JAVA_HOME to locate the correct Java Development Kit (JDK). The example uses Java 19.

```bash
export JAVA_HOME=/usr/lib/jvm/java-19-openjdk
```

--------------------------------

### Create Test Resources Script (Bash)

Source: https://openproblems.bio/documentation/create_task/dataset_processor

This Bash script is a placeholder for generating test resources. It includes commands to navigate to the repository root and provides an echo statement indicating that the actual commands for generating example files need to be implemented. It exits with an error code until implemented.

```bash
#!/bin/bash

# get the root of the directory
REPO_ROOT=$(git rev-parse --show-toplevel)

# ensure that the command below is run from the root of the repository
cd "$REPO_ROOT"

# remove this when you have implemented the script #<1>
echo "TODO: replace the commands in this script with the sequence of components that you need to run to generate test_resources." #<1>
echo "  Inside this script, you will need to place commands to generate example files for each of the 'src/api/file_*.yaml' files." #<1>
exit 1 #<1>

set -e

RAW_DATA=resources_test/common
DATASET_DIR=resources_test/task_template

mkdir -p $DATASET_DIR

```

--------------------------------

### Run Dataset Loader Component with Viash

Source: https://openproblems.bio/documentation/advanced_topics/create_a_dataset_loader

Command to execute the dataset loader component using Viash. It specifies the configuration file and the output file path for the generated dataset.

```bash
viash run src/datasets/loaders/myloader/config.vsh.yaml -- \
  --output mydataset.h5ad
```

--------------------------------

### Example Git Branch Naming for Hotfixes

Source: https://openproblems.bio/documentation/reference/naming

Demonstrates naming a git branch for a hotfix, emphasizing the 'hotfix' category and 'no-ref' when a specific issue is not applicable.

```shell
git branch hotfix/no-ref/registration-form-not-working
```

--------------------------------

### Rebuild Docker Container (Shell)

Source: https://openproblems.bio/documentation/create_component/add_a_control

Command to rebuild the Docker container after making changes to the component's dependencies in the config file. It uses 'viash run' with the '--setup cachedbuild' argument.

```shell
#| echo: false
viash run src/control_methods/my_python_method/config.vsh.yaml -- \
  ---setup cachedbuild

```

--------------------------------

### Run Viash Component with Help Argument

Source: https://openproblems.bio/documentation/fundamentals/common_commands

Executes a Viash component and displays its help text. This is achieved by passing the '--help' argument after the component configuration path.

```shell
viash run path/to/config.vsh.yaml -- --help
```

--------------------------------

### Build All Components with Viash

Source: https://openproblems.bio/documentation/fundamentals/common_commands

Builds all components within a specified namespace using Viash. The '--parallel' flag enables parallel execution for faster builds.

```shell
viash ns build --parallel
```

--------------------------------

### Sync Fork with Upstream Main Branch (Git)

Source: https://openproblems.bio/documentation/create_component/create_pull_request

Commands to update your local fork with the latest changes from the upstream repository's main branch. This ensures your contributions are based on the most current code.

```git
git remote add upstream <original repository URL>
git fetch upstream
git checkout main
git merge upstream/main
```

--------------------------------

### Download Test Data with Shell Script

Source: https://openproblems.bio/documentation/fundamentals/common_commands

This script downloads all test resources from an S3 bucket and stores them locally under the 'resources_test/' directory. It requires no specific arguments.

```shell
scripts/create_datasets/test_resources.sh
```

--------------------------------

### Create Directory for Dataset Loader

Source: https://openproblems.bio/documentation/advanced_topics/create_a_dataset_loader

Command to create a new directory for a custom dataset loader. This is the first step in organizing the loader's files.

```bash
mkdir src/datasets/loaders/myloader
```

--------------------------------

### Example Git Branch Naming for Features

Source: https://openproblems.bio/documentation/reference/naming

Illustrates how to name a git branch when adding, refactoring, or removing a feature, including a category, issue reference, and a descriptive kebab-cased name.

```shell
git branch feature/issue-42/create-new-button-component
```

--------------------------------

### Assertion Error: Missing Configuration Field

Source: https://openproblems.bio/documentation/create_component/run_tests

This example shows an AssertionError during configuration checking, specifically indicating that a 'summary' field is missing or empty. It highlights the importance of including all required fields in component configurations.

```python
+/tmp/viash_test_knn12945373156205296243/test_check_method_config/test_executable
Load config data
check general fields
Traceback (most recent call last):
Check info fields
  File "/viash_automount/tmp/viash_test_knn12945373156205296243/test_check_method_config/tmp//viash-run-knn-Xn2Vd7.py", line 42, in <module>
    assert "summary" in info is not None, "summary not an info field or is empty"
AssertionError: summary not an info field or is empty
```

--------------------------------

### Example Git Branch Naming for Bug Fixes

Source: https://openproblems.bio/documentation/reference/naming

Shows the correct format for naming a git branch when fixing a bug, incorporating the 'bugfix' category, issue number, and a description of the fix.

```shell
git branch bugfix/issue-342/button-overlap-form-on-mobile
```

--------------------------------

### Configure Dataset Processor (Viash YAML)

Source: https://openproblems.bio/documentation/create_task/dataset_processor

This snippet shows the Viash configuration for the 'process_dataset' component. It merges with a base interface, defines resources like a Python script and helper functions, specifies a Docker engine for reproducibility, and sets up runners.

```yaml
 __merge__: ../../api/comp_data_processor.yaml 
name: process_dataset 
resources:
  - type: python_script
    path: script.py
  - path: /common/helper_functions/subset_h5ad_by_format.py

engines:
  - type: docker
    image: openproblems/base_python:1.0.0

runners:
  - type: executable
  - type: nextflow
    directives:
      label: [highmem, highcpu, hightime]

```

--------------------------------

### Define Solution File Format (YAML)

Source: https://openproblems.bio/documentation/create_task/design_api

Defines the structure and metadata for a solution file, typically an AnnData file (.h5ad), used in Viash workflows. It specifies the file type, an example path, a label, a summary, and expected data slots.

```yaml
type: file
example: "resources_test/<task_id>/cxg_mouse_pancreas_atlas/solution.h5ad"
label: Solution
summary: "FILL IN: what this file represents"
info:
  format:

```

--------------------------------

### Test Component (Shell)

Source: https://openproblems.bio/documentation/create_component/add_a_control

Command to run unit tests for a component using 'viash test'. This helps verify the component's functionality and output format against the specified configuration file.

```shell
#| echo: false
# use knn instead of 'my_method' because the script won't work.
# maybe copy a 'working' script here
viash test src/control_methods/true_labels/config.vsh.yaml

```