TaskFarmer

https://lesterhedges.net/code/taskfarmer
Admin
Execute a list of system commands from a task file one-by-one, allowing many simulations to run...
Tokens:1,372
Snippets:26
Trust Score:7.3
Update:4 months ago
Show doc for...
Tokens:
Raw
### Compile and Install TaskFarmer

Source: https://lesterhedges.net/code/taskfarmer

Builds TaskFarmer and installs the executable and man page. Ensure Open MPI is installed prior to execution.

```bash
make
sudo make install
```

--------------------------------

### Running TaskFarmer with a Task File

Source: https://lesterhedges.net/code/taskfarmer

Example of preparing a task file and running TaskFarmer with 4 processes.

```bash
shuf examples/commands.txt | head -n 100 > tasks.txt | mpirun -np 4 taskfarmer -f tasks.txt
```

--------------------------------

### Obfuscated String Transformation

Source: https://lesterhedges.net/code/taskfarmer

An example of a shell command that transforms a string using `tr` for obfuscation or character substitution.

```bash
echo xyjw.mjiljx@lrfnq.h | tr "f-za-e" "a-ef-z"
```

--------------------------------

### Introduce Task Dependencies

Source: https://lesterhedges.net/code/taskfarmer

Create task dependencies by placing multiple tasks on a single line, separated by semicolons. This ensures tasks execute in a specific order.

```bash
perform_calculation > data.txt; analyze_data < data.txt
```

--------------------------------

### TaskFarmer Command-Line Options

Source: https://lesterhedges.net/code/taskfarmer

This shows the general command-line structure for TaskFarmer, including required and optional arguments.

```bash
mpirun -np CORES taskfarmer [-h] -f FILE [-v] [-w] [-r] [-s SLEEP_TIME]  
[-m MAX_RETRIES]
```

--------------------------------

### Count Remaining Tasks with wc

Source: https://lesterhedges.net/code/taskfarmer

Use the `wc -l` command to count remaining tasks in task files. This is useful for monitoring progress without parsing TaskFarmer logs.

```bash
wc -l task_files/*
```

--------------------------------

### Disable InfiniBand Fork Support in OpenMPI

Source: https://lesterhedges.net/code/taskfarmer

Workaround for InfiniBand interconnects causing issues with `fork()` in OpenMPI. Set environment variables to disable InfiniBand support for fork.

```bash
export OMPI_MCA_mpi_warn_on_fork=0
```

```bash
export OMPI_MCA_btl_openib_want_fork_support=0
```

--------------------------------

### Compile TaskFarmer with a Different Compiler

Source: https://lesterhedges.net/code/taskfarmer

Builds TaskFarmer using a specified compiler, such as 'cc' for Cray systems. This overrides the default compiler setting in the Makefile.

```bash
make CC=cc
```

--------------------------------

### Redirect Task Output to Log File

Source: https://lesterhedges.net/code/taskfarmer

Redirect standard output of system commands within a task file to a separate log file. This prevents cluttering the main TaskFarmer output.

```bash
echo "Hello, I'm a task" > job.log
```

--------------------------------

### Append Tasks to Task File

Source: https://lesterhedges.net/code/taskfarmer

Safely modify the task file externally by appending tasks using redirection. This avoids conflicts with TaskFarmer's I/O, especially when simulations are short.

```bash
cat more_tasks >> tasks.txt
```

--------------------------------

### Remove TaskFarmer

Source: https://lesterhedges.net/code/taskfarmer

Completely removes TaskFarmer from the system.

```bash
sudo make uninstall
```
Tokens:
Raw