### Template Job File Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Provides a concrete example of a job file ('template.example') that defines job parameters and commands. This file is parsed by 'qmap submit' to configure and launch jobs.

```text
submit-example

Job pre-commands
  echo "Starting job"

Job parameters
  memory 128MB
  cores 1

Job command
  sleep 60

Job post-commands
  echo "Job finished"
```

--------------------------------

### Profile Configuration Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/concepts.rst

This example demonstrates how to configure a profile for the QMAP executor, specifying the executor type and default job parameters. It shows how to set parameters like cores, memory, and time, as well as editable parameters.

```ini
executor = slurm

[params]
cores = 7

[editable_params]
cores = Cores
```

--------------------------------

### Basic qmap template Usage

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

This demonstrates the fundamental usage of the 'qmap template' command with a simple command string. It shows how the tool can prepend environment setup commands (like module loads and source activations) and the core job command to the output.

```sh
qmap template "<command>" -m <memory> -c <cores> -o <output file>
```

--------------------------------

### qmap Memory Jobs File Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

An example 'jobs' file demonstrating memory requirements for different jobs. This file format allows users to specify resource needs for each task submitted via qmap.

```text
script: test.py
mem: 512
script: test2.py
mem: 1024
```

--------------------------------

### Job File Format Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Illustrates the structure of a job file used by 'qmap submit'. It shows sections for job pre-commands, job parameters, job commands, and job post-commands, as well as how to specify resources.

```text
# Job pre-commands
# Command to be executed before any job

# Job parameters
# Resources asked to the workload manager (e.g. memory or cores)

# Job command
# Bash command to be executed.
# One command corresponds to one job unless :ref:`groups <grouping>` are made

# Job post-commands
# Commands to be executed before any job

```

--------------------------------

### qmap Jobs File Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

A sample 'jobs' file format used by qmap for defining job submission parameters. This file typically specifies scripts to run and their arguments.

```text
script: hello.py
arg: "hello world"
arg: --test
```

--------------------------------

### qmap template: Auto-detect EasyBuild and Conda

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

This example illustrates how 'qmap template' automatically detects and includes relevant 'module load' commands for EasyBuild and 'source activate' commands for conda environments when generating a jobs file. This saves users from manually specifying these setup steps.

```console
$ qmap template "sleep 5"
[pre]
module load anaconda3/4.4.0
source activate test_qmap

[jobs]
sleep 5
```

--------------------------------

### qmap submit Command Example with Environment Variables

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Demonstrates how to use 'qmap submit' with command-line parameters and how job parameters can be accessed within job commands via environment variables. This allows for dynamic script generation based on submitted resources.

```bash
qmap submit --cores 5 jobs.map
```

```bash
python parallel1.py --cores ${QMAP_CORES}
python parallel2.py --cores ${QMAP_CORES}
python parallel3.py --cores ${QMAP_CORES}
```

--------------------------------

### Example: Get Job Metadata Fields

Source: https://github.com/bbglab/qmap/blob/master/docs/source/info.rst

An example demonstrating how to retrieve specific metadata fields ('usage.time.elapsed' and 'retries') for jobs with a 'completed' status. The output includes the job ID and the requested fields, with empty strings for any missing data.

```console
$ qmap info -s completed usage.time.elapsed retries
id      usage.time.elapsed      retries
1       00:00:12        0
0       00:00:07        0
```

--------------------------------

### qmap Output File Structure Example

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Illustrates the typical file structure generated in the output directory after a qmap job submission. Includes job-specific error (.err), info (.info), output (.out), and script (.sh) files, along with the input file copy (qmap_input).

```console
$ ls hello_20180905
0.err  0.info  0.out  0.sh  1.err  1.info  1.out  1.sh  qmap_input default.env
```

--------------------------------

### qmap template: Include Job Parameters

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

This example shows how 'qmap template' can incorporate job-specific parameters like memory ('-m') and cores ('-c') into the generated jobs file. These parameters are logged under a '[params]' section, providing clear visibility of resource allocation.

```console
$ qmap template "sleep 5" -c 1 -m 1G
[params]
memory=1G
cores=1

[jobs]
sleep 5
```

--------------------------------

### QMap Executor: Direct Cluster Interaction (Python)

Source: https://context7.com/bbglab/qmap/llms.txt

Provides Python examples for using the QMap executor to interact directly with HPC clusters. It covers loading different executor types (slurm, sge, lsf, local, dummy), creating job scripts, submitting jobs with specified parameters, monitoring job statuses, cancelling jobs, running commands interactively, and retrieving cluster usage statistics.

```python
from qmap import executor
from qmap.job.parameters import Parameters

# Load executor (done automatically by Profile)
executor.load('slurm', show_usage=False)  # slurm, sge, lsf, local, dummy

# Create job script
executor.create_script(
    file='job_script',  # Creates job_script.sh
    commands=['module load python', 'python analyze.py'],
    default_params_file='default.env',
    specific_params_file='specific.env'
)

# Submit job
params = Parameters({'cores': 8, 'memory': '32G', 'time': '4h', 'name': 'analysis_job'})
job_id, submit_cmd = executor.run_job(
    f_script='job_script',
    job_parameters=params,
    out='job.out',
    err='job.err'
)
print(f"Submitted job {job_id} with command: {submit_cmd}")

# Get status for multiple jobs
job_ids = ['12345', '12346', '12347']
for job_id, (status, error, info) in executor.generate_jobs_status(job_ids):
    print(f"Job {job_id}: {status}")
    print(f"  Error code: {error}")
    print(f"  Memory used: {info['usage'].get('memory', 'N/A')}")
    print(f"  Time elapsed: {info['usage'].get('time', 'N/A')}")

# Cancel jobs
executor.terminate_jobs(['12345', '12346'])

# Run command directly (interactive execution)
executor.run('python script.py', params, quiet=False)

# Get cluster usage statistics (if supported)
usage = executor.get_usage()
print(f"Nodes in use: {usage['nodes']}")
print(f"Cluster usage: {usage['usage']}%")
print(f"User usage: {usage['user']}%")
```

--------------------------------

### Example: Reattach Qmap Without Console Output (Console)

Source: https://github.com/bbglab/qmap/blob/master/docs/source/reattach.rst

This console example demonstrates the execution of 'qmap reattach --no-console'. The '--no-console' flag suppresses any console output during the reattachment process. The output indicates the successful completion of the manager process.

```console
$ qmap reattach --no-console
Finished vs. total: [2/2]
Manager finished
```

--------------------------------

### Jobs Map File Format: Basic structure

Source: https://context7.com/bbglab/qmap/llms.txt

Defines the basic structure of a QMAP jobs map file using INI format. It includes sections for pre-execution commands, post-execution commands, default parameters, and the job commands themselves, with examples of inline parameter overrides.

```ini
[pre]
module load bioinfo-tools
module load samtools/1.9
conda activate myenv

[post]
echo "Job completed"

[params]
cores = 8
memory = 16G
time = 2h

[jobs]
bwa mem reference.fa sample1_R1.fq sample1_R2.fq | samtools view -bS - > sample1.bam
bwa mem reference.fa sample2_R1.fq sample2_R2.fq | samtools view -bS - > sample2.bam
samtools sort sample1.bam -o sample1.sorted.bam
samtools sort sample2.bam -o sample2.sorted.bam

# Inline job-specific parameters override defaults
process_large.sh ## cores=16 memory=64G
```

--------------------------------

### QMAP CLI: Get resource usage for completed jobs

Source: https://context7.com/bbglab/qmap/llms.txt

This command retrieves resource usage metrics (time, memory, disk I/O, nodes) for jobs that have completed successfully. It requires specifying the log directory and the job status filter.

```bash
qmap info -l logs/ -s completed \
  command \
  usage.time \
  usage.memory \
  usage.disk.read \
  usage.disk.write \
  usage.cluster.nodes
```

--------------------------------

### Expand Command Patterns Manually and from File

Source: https://context7.com/bbglab/qmap/llms.txt

Demonstrates how to expand command patterns, either directly provided as strings or read from a file. Supports basic templating with curly braces and glob expansion. The `expand_command` and `expand` functions are used.

```python
# Expand command patterns manually
commands = list(expand_command('process {{file1,file2,file3}}.txt'))
# Result: ['process file1.txt', 'process file2.txt', 'process file3.txt']

# Expand from file (one parameter per line)
commands = list(expand('samples.txt'))  # If file exists, reads lines
commands = list(expand('analyze {{*.csv}}'))  # Glob expansion
```

--------------------------------

### qmap Submit Help Command

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Displays the help information for the 'qmap submit' command, listing all available options and their descriptions. This is essential for understanding the full range of customization for job submissions.

```sh
qmap submit --help
```

--------------------------------

### Save Jobs and Environment Files using QMap

Source: https://context7.com/bbglab/qmap/llms.txt

Demonstrates how to save job configuration files and environment parameters using the QMap library. This involves saving job configurations to a specified directory and saving/loading environment parameters with a defined set of properties.

```python
jobs_file.save('original.qmap', 'logs/')  # Copies to logs/qmap_input

env_file.save('job.env', Parameters({'cores': 8, 'memory': '32G'}))
params = env_file.load('job.env')
```

--------------------------------

### Create Job Template File (Bash)

Source: https://github.com/bbglab/qmap/blob/master/README.rst

Generates a jobs map file for use with 'qmap submit'. It supports various wildcard expansions and environment module loading. The file format includes commands to be executed before and after each job.

```bash
qmap template "<command with wildcards>" -f <jobs map file>
```

--------------------------------

### Generate Jobs File from Command Pattern

Source: https://context7.com/bbglab/qmap/llms.txt

Generates a complete jobs file by processing a command string with specified parameters and environment settings. It uses the `generate` function and writes the output to a '.qmap' file. Dependencies include the `qmap.job.parameters` module.

```python
from qmap.job.parameters import Parameters

jobs_content = generate(
    command='python process.py --input {{sample1,sample2,sample3}} --cores ${QMAP_CORES}',
    job_parameters=Parameters({'cores': 4, 'memory': '16G', 'time': '2h'}),
    base_template=None,  # Optional: path to base template
    conda_env='bioinfo',
    easy_build_modules=['bioinfo-tools', 'samtools/1.9']
)
with open('generated.qmap', 'w') as f:
    f.write('\n'.join(jobs_content))
```

--------------------------------

### Python API: Template generation for dynamic job creation

Source: https://context7.com/bbglab/qmap/llms.txt

Introduces QMAP's template generation capabilities for creating job files dynamically. This involves wildcard expansion and parameter substitution, facilitated by functions like `generate`, `expand_command`, and `expand`.

```python
from qmap.template import generate, expand_command, expand
from qmap.job.parameters import Parameters

# Placeholder for actual template generation code
```

--------------------------------

### Submit qmap Job with Memory and Cores

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Submits a qmap job with specified memory and core requirements. Requires a jobs file as input. The output logs are stored in .out and .err files.

```sh
qmap submit -m <memory> -c <cores> <jobs file>
```

--------------------------------

### Execute Command with Resources using qmap run

Source: https://github.com/bbglab/qmap/blob/master/docs/source/run.rst

This command executes a specified command on a cluster with user-defined memory and core allocations. It's useful for running resource-intensive tasks efficiently and provides job statistics after execution.

```sh
qmap run -m <memory> -c <cores> "<command>"
```

```console
$ qmap run -c 6 -m 12G "sleep 5 && echo 'hello world'"
Executing sleep 5 && echo 'hello world'
salloc: Granted job allocation 31707
hello world
salloc: Relinquishing job allocation 31707
Elapsed time:  00:00:05
Memory  0G
```

```console
$ qmap run -m 12 "python test/python_scripts/memory.py 10"
Executing python test/python_scripts/memory.py 10
salloc: Granted job allocation 36015
1 Gb
...
10 Gb
salloc: Relinquishing job allocation 36015
Elapsed time:  00:00:36
Memory  10G
```

--------------------------------

### Expand Command Patterns with Named Wildcards

Source: https://context7.com/bbglab/qmap/llms.txt

Illustrates the use of named wildcards in command patterns to capture and reuse specific parts of the expanded command. This allows for more dynamic and structured command generation. The `expand_command` function is utilized.

```python
# Named wildcards - capture and reuse values
pattern = 'bwa mem {{?ref:reference*.fa}} reads.fq > {{?=ref}}.sam'
for cmd in expand_command(pattern):
    print(cmd)
# Result: bwa mem reference_hg38.fa reads.fq > reference_hg38.sam
```

--------------------------------

### qmap template: User Wildcards with Files

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

Illustrates using a file name within user wildcards '{{...}}'. 'qmap template' reads each line from the specified file and generates a separate job entry for each line, treating each line as a value for the wildcard.

```console
$ qmap template "sleep {{sleep_times.txt}}"
[jobs]
sleep 5
sleep 10
```

--------------------------------

### qmap template: User Wildcards with Lists

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

Demonstrates the use of user-defined wildcards, indicated by '{{...}}', to specify a list of values. 'qmap template' expands the wildcard into multiple job entries, each using one item from the provided comma-separated list.

```console
$ qmap template "sleep {{5,10}}"
[jobs]
sleep 5
sleep 10
```

--------------------------------

### File Operations - Job Metadata and Parsing

Source: https://context7.com/bbglab/qmap/llms.txt

Details how to work with job metadata files and parse job map files using qmap. This includes saving and loading job metadata, retrieving specific fields from multiple metadata files, parsing '.qmap' files into commands and parameters, and filtering jobs by status. Dependencies: `qmap.file.metadata`, `qmap.file.jobs`, `qmap.job.JobStatus`.

```python
from qmap.file import metadata as metadata_file
from qmap.file import jobs as jobs_file
from qmap.file import env as env_file
from qmap.job import JobStatus

# Save job metadata
metadata = {
    'command': 'python analyze.py',
    'status': JobStatus.DONE,
    'executor_id': '12345',
    'retries': 0,
    'exit_code': 0,
    'usage': {
        'time': '01:23:45',
        'memory': '8.5G',
        'disk': {'read': '1.2G', 'write': '500M'}
    }
}
metadata_file.save('logs/job_001.info', metadata)

# Load job metadata
metadata = metadata_file.load('logs/job_001.info')
print(f"Command: {metadata['command']}")
print(f"Exit code: {metadata['exit_code']}")

# Get fields from multiple metadata files
for line in metadata_file.get_fields('logs/', ['command', 'status', 'usage.memory'], [JobStatus.FAILED]):
    print(line)

# Parse jobs file
pre_cmds, job_cmds, post_cmds, params = jobs_file.parse('jobs.qmap')
print(f"Pre-commands: {pre_cmds}")
print(f"Job commands: {list(job_cmds.items())}")
print(f"Post-commands: {post_cmds}")
print(f"Parameters: {params}")

# Filter jobs by status
sections = jobs_file.filter_commands('logs/', [JobStatus.FAILED], only_jobs=True)
for line in jobs_file.stringify(sections):
    print(line)
```

--------------------------------

### Execute Command with Resources (Bash)

Source: https://github.com/bbglab/qmap/blob/master/README.rst

Runs a command with specified memory and core resources, preserving the working environment. This is a core utility for executing individual jobs.

```bash
qmap run -m <memory> -c <cores>  "<command>"
```

--------------------------------

### Profile Management - Executor Configuration

Source: https://context7.com/bbglab/qmap/llms.txt

Demonstrates how to manage execution profiles in qmap, allowing configuration of different cluster environments and settings. Profiles can be loaded from built-in types, custom files, or configuration namespaces. It covers accessing and modifying job parameters associated with a profile. Dependencies: `qmap.profile.Profile`, `qmap.job.parameters.Parameters`.

```python
from qmap.profile import Profile
from qmap.job.parameters import Parameters

# Load built-in profile
profile = Profile('slurm')  # Built-in: slurm, sge, lsf, local

# Load custom profile from file
profile = Profile('/path/to/custom_profile.conf')

# Load profile from config namespace (searches ~/.config/qmap/)
profile = Profile('my_cluster_config')

# Default local execution
profile = Profile()  # Uses local executor

# Access profile configuration
print(f"Executor: {profile['executor']}")
print(f"Show usage: {profile.get('show_usage', False)}")
print(f"Max ungrouped jobs: {profile.get('max_ungrouped', float('inf'))}")

# Access and modify job parameters
params = profile.parameters
print(f"Default cores: {params.get('cores', 1)}")
print(f"Default memory: {params.get('memory', '4G')}")

# Update parameters
params.update(cores=16, memory='64G', time='8h')

# Profile dictionary structure
profile_dict = {
    'executor': 'slurm',
    'show_usage': True,
    'max_ungrouped': 1000,
    'params': {'cores': 8, 'memory': '32G', 'time': '4h'},
    'editable_params': {'cores': 'Cores', 'memory': 'Memory', 'time': 'Time'}
}
profile = Profile(profile_dict)
```

--------------------------------

### Complex Command Expansion with Multiple Wildcards

Source: https://context7.com/bbglab/qmap/llms.txt

Shows how to handle complex command expansions involving multiple wildcards, including named wildcards for input and output processing. This enables flexible generation of commands for data processing pipelines. `expand_command` is the primary function used.

```python
# Complex expansion with multiple wildcards
pattern = 'process --input {{?in:data/*.csv}} --output results/{{?=in}}'
for cmd in expand_command(pattern):
    print(cmd)
```

--------------------------------

### QMAP CLI: Create job file from failed jobs

Source: https://context7.com/bbglab/qmap/llms.txt

This command generates a QMAP job file containing details of all failed jobs found in the specified log directory. The output is redirected to a file named 'failed_jobs.qmap'.

```bash
qmap info -l logs/ -s failed > failed_jobs.qmap
```

--------------------------------

### qmap template: Glob Wildcards

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

Shows the application of glob wildcards ('*' and '**') within the command string. 'qmap template' utilizes Python's glob module to expand these wildcards, generating job entries for all matching files or paths.

```console
$ qmap template "mypgrog --input *.txt"
[jobs]
mypgrog --input file1.txt
mypgrog --input file2.txt
```

--------------------------------

### Execute Single Command with Resources (Bash)

Source: https://context7.com/bbglab/qmap/llms.txt

The `qmap run` command executes a single command with specified computational resources (cores, memory) while preserving the current working environment. It supports custom profiles for different workload managers and can run quietly to suppress qmap output.

```bash
# Run a memory-intensive command with 8 cores and 32GB RAM
qmap run -c 8 -m 32G "python process_data.py --input large_dataset.csv"

# Run with custom profile (Slurm configuration)
qmap run -p slurm -c 16 -m 64 -q "samtools sort -@ 16 input.bam -o sorted.bam"

# Execute quietly (suppress qmap output, show only command output)
qmap run -c 4 -m 8G --quiet "bwa mem reference.fa reads.fq > aligned.sam"
```

--------------------------------

### Query Job Metadata with qmap info

Source: https://github.com/bbglab/qmap/blob/master/docs/source/info.rst

This command retrieves specified fields from job metadata. It takes a status, a qmap logs folder, and a list of fields as arguments. Missing fields will result in an empty string in the output. The output is tab-separated by default, or pipe-separated if the collapse flag is used.

```sh
qmap info -s <status> -l <qmap logs folder> <field 1> <field 2> ... <field n>
```

--------------------------------

### Generate Job Map File from Command Patterns (Bash)

Source: https://context7.com/bbglab/qmap/llms.txt

The `qmap template` command generates a job map file by expanding wildcards and substitutions from command patterns. It supports various input formats, including file lists, comma-separated parameters, and named wildcards. It can also specify conda environments and EasyBuild modules.

```bash
# Create jobs for all CSV files in directory
qmap template "python analyze.py --input {{*.csv}}" -o jobs.qmap

# Generate jobs from comma-separated list with parameters
qmap template "process_sample.sh {{sample1,sample2,sample3}}" -o batch.qmap -c 4 -m 16G

# Use file with list of parameters (one per line)
qmap template "run_analysis.py --id {{samples.txt}}" -o analysis.qmap

# Named wildcards - use value in multiple places
qmap template "bwa mem {{?ref:*.fa}} reads.fq | samtools view -o {{?=ref}}.bam" -o align.qmap

# Specify conda environment and EasyBuild modules
qmap template "python script.py {{input/*.txt}}" -o jobs.qmap \
  --conda-env myenv \
  --module bioinfo-tools \
  --module samtools/1.9

# Use base template with pre-configured settings
qmap template "process.sh {{data*.txt}}" -o jobs.qmap -b base_template.qmap
```

--------------------------------

### qmap template: Named Wildcards

Source: https://github.com/bbglab/qmap/blob/master/docs/source/template.rst

Demonstrates the advanced 'named groups' feature where wildcards are assigned names (e.g., '{{?f_name:*}}') and can be referenced elsewhere using '{{?=f_name}}'. This allows for complex substitutions and combinatorial generation of jobs based on named parameters.

```console
$ qmap template "myprog --input {{?f_name:*}}.txt --variable {{?v_name:a,b}} --output {{?=f_name}}_{{?=v_name}}"
[jobs]
myprog --input file1.txt --variable a --output file1_a
myprog --input file2.txt --variable a --output file2_a
myprog --input file1.txt --variable b --output file1_b
myprog --input file2.txt --variable b --output file2_b
```

--------------------------------

### qmap Submit with Default Working Directory (Failure)

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Illustrates a qmap job submission without specifying a working directory, resulting in job failures. This scenario highlights the importance of the '-w' flag when scripts are not in the default location.

```console
$ qmap submit memory.jobs --no-console
Finished vs. total: [0/2]
Job 2 failed. [1/2]
Job 3 failed. [2/2]
Manager finished
```

--------------------------------

### Status Tracking - Monitor Execution State

Source: https://context7.com/bbglab/qmap/llms.txt

Explains how to use the `Status` object to track the execution state of jobs. It allows grouping jobs by their status (unsubmitted, running, pending, completed, failed) and updating their states from the cluster. Dependencies: `qmap.manager.Status`, `qmap.job.JobStatus`.

```python
from qmap.manager import Status
from qmap.job import JobStatus

# Status object (typically created by Manager)
status = Status(jobs_dict)  # jobs_dict: {id: Job}

# Access job groups by status
unsubmitted_ids = status.groups[JobStatus.UNSUBMITTED]
running_ids = status.groups[JobStatus.RUN]
pending_ids = status.groups[JobStatus.PENDING]
completed_ids = status.groups[JobStatus.DONE]
failed_ids = status.groups[JobStatus.FAILED]
other_ids = status.groups[JobStatus.OTHER]

# Update all job statuses from cluster
status.update()  # Queries cluster for job states

# Status automatically tracks changes via job state handlers
print(status)
# Output: Unsubmitted: 10   Run: 5   Pending: 3   Done: 82   Failed: 0   Other: 0   TOTAL: 100

# Manual notification (typically called by Job objects)
status.notify_state_change('job_001', JobStatus.PENDING, JobStatus.RUN)

# Total jobs
print(f"Total jobs: {status.total}")
```

--------------------------------

### qmap Submit with Working Directory

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Submits a qmap job specifying a working directory using the '-w' flag. This is useful when job scripts are not located in the current directory. Shows successful job completion after setting the working directory.

```console
$ qmap submit memory.jobs -w test/python_scripts/ --no-console
Finished vs. total: [0/2]
Job 2 done. [1/2]
Job 3 done. [2/2]
Manager finished
```

--------------------------------

### Python API: Manager classes for job execution

Source: https://context7.com/bbglab/qmap/llms.txt

Demonstrates using QMAP's Manager classes (Submitted, Reattached) to control job execution programmatically. It covers creating, monitoring, updating, resubmitting, and terminating jobs, as well as reattaching to existing executions.

```python
from qmap.manager import Submitted, Reattached
from qmap.profile import Profile
from qmap.job import JobStatus, JobParameters
import time

# Create new execution from jobs file
profile = Profile('slurm')  # or Profile() for local, Profile('/path/to/profile.conf')
manager = Submitted(
    jobs_file='analysis.qmap',
    output_folder='logs_20250108',
    profile_conf=profile,
    max_running_jobs=10,
    group_size=None,  # or integer to group commands
    cli_params=JobParameters({'cores': 8, 'memory': '32G', 'time': '4h'})
)

# Monitor execution
while not manager.is_done:
    manager.update()  # Updates job statuses and submits new jobs
    print(manager.status)  # Shows: Unsubmitted: X Run: Y Pending: Z Done: W Failed: V
    time.sleep(30)

# Get jobs by status
unsubmitted = manager.get_jobs(JobStatus.UNSUBMITTED)
running = manager.get_jobs(JobStatus.RUN)
failed = manager.get_jobs(JobStatus.FAILED)

# Resubmit failed jobs with updated parameters
manager.resubmit_failed(cores=16, memory='64G')

# Update job parameters for future submissions
manager.update_job_params(cores=12, memory='48G')

# Gracefully close (save metadata)
manager.close()

# Force submit all remaining jobs before closing
manager.submit_and_close()

# Cancel all running/pending jobs
manager.terminate()

# Access specific job
job = manager.get('0000')
print(f"Job {job.id}: {job.command} - Status: {job.status}")

# Reattach to previous execution
manager = Reattached(
    output_folder='logs_20250108',
    force=False,  # Set True to load despite errors
    max_running_jobs=20  # Optional: change limit
)

# Continue monitoring
manager.is_submission_enabled = True  # Allow submitting unsubmitted jobs
while not manager.is_done:
    manager.update()
    print(manager.status)
    time.sleep(30)
```

--------------------------------

### Submit and Manage Job Execution (Bash)

Source: https://context7.com/bbglab/qmap/llms.txt

The `qmap submit` command submits jobs defined in a map file, controlling parallelism and monitoring execution. It allows setting maximum running jobs, grouping commands per job, specifying time limits, custom job names, and profiles. It also supports non-interactive console usage and specifying working directories.

```bash
# Submit jobs with max 10 running simultaneously
qmap submit jobs.qmap --logs output_logs --max-running 10 -c 4 -m 8G

# Group 5 commands per job (efficient for small tasks)
qmap submit jobs.qmap -l logs/ -r 20 -g 5

# Submit with time limit and custom job name prefix
qmap submit analysis.qmap -l results/ -r 5 -t 2h -n myproject

# Use non-interactive console (for screen/tmux sessions)
qmap submit jobs.qmap -l logs/ -r 10 --no-console

# Submit with profile and working directory
qmap submit jobs.qmap -l logs/ -r 15 -p slurm -w /scratch/project/

# Full example with all resource parameters
qmap submit pipeline.qmap \
  --logs execution_logs_20250108 \
  --max-running 50 \
  --cores 8 \
  --memory 32G \
  --time 4h \
  --name genomics_batch \
  --profile slurm

# Job commands can use special variables:
# ${QMAP_LINE} - unique job identifier
# ${QMAP_CORES} - number of cores allocated
# Example in jobs.qmap:
# [jobs]
# echo "Job ${QMAP_LINE} using ${QMAP_CORES} cores"
```

--------------------------------

### Python API: Job classes for individual task control

Source: https://context7.com/bbglab/qmap/llms.txt

Illustrates the use of QMAP's Job classes (SubmittedJob, ReattachedJob) for managing individual tasks. This includes programmatic job creation, submission, status checking, resubmission with new parameters, metadata saving/loading, and memory unit conversion.

```python
from qmap.job import SubmittedJob, ReattachedJob, JobStatus
from qmap.job.parameters import Parameters, memory_convert

# Create a job programmatically
job = SubmittedJob(
    id_='job_001',
    folder='output_logs/',
    command='python analyze.py --input data.csv',
    parameters=Parameters({'cores': 8, 'memory': '32G'})
    pre_commands=['module load python/3.9', 'conda activate analysis'],
    post_commands=['echo "Analysis complete"']
)

# Submit job for execution
job.run(default_params)  # Merges with job-specific parameters
print(f"Job submitted with executor ID: {job.executor_id}")

# Check status
job.update()
print(f"Status: {job.status}")

# Resubmit failed job with different parameters
if job.status == JobStatus.FAILED:
    job.resubmit(parameters=Parameters({'cores': 16, 'memory': '64G'}))
    job.run(default_params)

# Save metadata to disk
job.save_metadata()

# Access job information
print(f"Command: {job.command}")
print(f"Retries: {job.retries}")
print(f"Stdout: {job.f_stdout}")
print(f"Stderr: {job.f_stderr}")
print(f"Metadata: {job.metadata}")

# Load existing job from disk
job = ReattachedJob(id_='job_001', folder='output_logs/')
print(f"Loaded job: {job.command} - Status: {job.status}")

# Memory unit conversion
mem_gb = memory_convert(32768, 'M', 'G')  # Convert 32768 MB to GB
mem_tb = memory_convert(2048, 'G', 'T')   # Convert 2048 GB to TB
print(f"{mem_gb}G, {mem_tb}T")
```

--------------------------------

### Explore Job Metadata (Bash)

Source: https://github.com/bbglab/qmap/blob/master/README.rst

Retrieves metadata information generated by 'qmap submit' for each job. It allows querying specific fields and filtering jobs by status. If no fields are provided, it returns the input commands for the jobs.

```bash
qmap info --logs <logs folder> <field 1> <field 2>.<subfield 1> ...
```

```bash
qmap info --logs <logs folder> --status <completed|failed|other|pending|running|unsubmitted|all>
```

--------------------------------

### Query Job Metadata and Status (Bash)

Source: https://context7.com/bbglab/qmap/llms.txt

The `qmap info` command queries job metadata and status from execution logs. It can display failed, completed, or all jobs, and extract specific metadata fields using dot notation for nested values. Results can be filtered by status and exported to a file.

```bash
# Show failed job commands (default behavior)
qmap info --logs output_logs

# Get failed job commands only (collapsed format)
qmap info -l logs/ -s failed --collapse

# Show completed jobs
qmap info -l logs/ -s completed

# Extract specific metadata fields
qmap info -l logs/ status command exit_code

# Get nested fields with dot notation
qmap info -l logs/ usage.memory usage.time usage.cluster.nodes

# Filter multiple statuses
qmap info -l logs/ -s failed -s running command status

# Export to file
qmap info -l logs/ -s all command status -o results.txt
```

--------------------------------

### Environment Variable for Profile

Source: https://github.com/bbglab/qmap/blob/master/docs/source/concepts.rst

This snippet shows how to set the QMAP_PROFILE environment variable, which specifies the profile to be used by QMAP. This can be a profile name from the default configuration directory or a direct path to a profile file.

```shell
# Example: setting QMAP_PROFILE to a profile file
export QMAP_PROFILE="~/.config/qmap/my_custom_profile"

# Example: setting QMAP_PROFILE to a profile name
export QMAP_PROFILE="slurm"
```

--------------------------------

### Submit Jobs from Map File (Bash)

Source: https://github.com/bbglab/qmap/blob/master/README.rst

Executes all jobs defined in a jobs map file. It provides controls for resource allocation, log management, and job submission limits to prevent overwhelming the workload manager. It also supports grouping small jobs and substituting job parameters.

```bash
qmap submit -m <memory> -c <cores>  <jobs map file> --logs <logs folder> --max-running <#>
```

--------------------------------

### qmap Error: Non-Empty Output Directory

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Demonstrates the error message when attempting to submit a qmap job to an output directory that already contains files. This prevents accidental overwriting and ensures data integrity.

```console
$ qmap submit hello.jobs --no-console
QMapError: Output folder [hello_20870905] is not empty. Please give a different folder to write the output files.
```

--------------------------------

### Reattach to QMap Execution (Bash)

Source: https://github.com/bbglab/qmap/blob/master/README.rst

Allows reattaching to a previous 'qmap submit' execution from its logs directory. It can submit any jobs that were not previously submitted and offers a 'no-console' interface.

```bash
qmap reattach --logs <logs folder>
```

--------------------------------

### Reconnect to Previous Execution (Bash)

Source: https://context7.com/bbglab/qmap/llms.txt

The `qmap reattach` command allows reconnecting to a previous job execution session to monitor its status or submit remaining jobs. It can be used in the current directory or by specifying a logs folder. Options include changing the maximum running jobs limit and forcing reattachment even with loading errors.

```bash
# Reattach to execution in current directory
qmap reattach

# Reattach to specific logs folder
qmap reattach --logs output_logs

# Reattach and change max running jobs limit
qmap reattach -l logs/ -r 20

# Force reattachment even with loading errors
qmap reattach -l logs/ --force

# Reattach with non-interactive console
qmap reattach -l logs/ --no-console
```

--------------------------------

### Filter Job Commands by Status with qmap info

Source: https://github.com/bbglab/qmap/blob/master/docs/source/info.rst

This command filters job commands based on their status. It is invoked when no fields are specified, and it requires a status and a qmap logs folder. The collapse flag, when used in this mode, removes empty lines from the output.

```sh
qmap info -s <status> -l <qmap logs folder>
```

--------------------------------

### Reattach Qmap Submission (Shell)

Source: https://github.com/bbglab/qmap/blob/master/docs/source/reattach.rst

This command reattaches to a previous qmap submission using the logs generated by 'qmap submit'. It can take an optional '--logs' argument to specify the folder containing the logs. Ensure the logs are accessible for successful reattachment.

```sh
qmap reattach [--logs <folder>]
```

--------------------------------

### qmap Submit with Job Grouping

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Submits qmap jobs with the '-g' flag for grouping. This reduces the number of individual job submissions but may ignore specific execution parameters for individual jobs within a group. Shows consolidated job completion.

```console
$ qmap submit hello.jobs -g 2 --no-console
Specific job execution parameters ignored
Finished vs. total: [0/1]
Job 0 done. [1/1]
Manager finished
```

--------------------------------

### Submit qmap Job with No Console Output

Source: https://github.com/bbglab/qmap/blob/master/docs/source/submit.rst

Submits a qmap job and suppresses console output using the '--no-console' flag. This is useful for automated runs where only log files are needed. Shows progress of job completion.

```console
$ qmap submit -m 1 -c 1 hello.jobs --no-console
Finished vs. total: [0/2]
Job 0 done. [1/2]
Job 1 done. [2/2]
Manager finished
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.