### Registry lookup and compute_advantage example
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-a3.md
Demonstrates how verl's actor retrieves the A3_algo algorithm from the registry and calls compute_advantage. Requires a tokenizer; missing tokenizer raises an error (see next example). The advantages are stored at data.batch["advantages"], and debug metrics include actor/A3_algo_tree_gate_mean.
```python
from main_entry.cli_agent_bash_coding.algo.registry import (
get_bash_coding_algorithm, list_bash_coding_algorithms,
)
assert "A3_algo" in list_bash_coding_algorithms()
algo = get_bash_coding_algorithm("A3_algo")
assert algo.name == "A3_algo"
data = algo.compute_advantage(
data=batch,
multi_turn=True,
epsilon=1e-6,
adv_bound=1.1,
trie_min_count=2,
w_trie=0.65,
support_prior=0.1,
decision_gamma=0.9,
w_action_cluster=1.0,
tokenizer=tokenizer,
intent_scopes=[1, 2, 3, -1],
w_intent_scopes=[0.25, 0.25, 0.25, 0.25],
A3_algo_tree_intent_match_tau=0.05,
)
adv = data.batch["advantages"] # [B, resp_len] token-level advantages
dbg = data.meta_info["algorithm_debug_metrics"]
print(dbg["actor/A3_algo_tree_gate_mean"])
```
--------------------------------
### Create conda environment and install CLI dependencies
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md
Creates the conda environment from environment.yml, activates it, and runs install_apt.sh to prepare apt libraries required for CLI-agent execution in the sandbox. The research implementation is docker-free.
```bash
conda env create -f environ/environment.yml
conda activate cli_agent
bash main_entry/pre_build/install_apt.sh
```
--------------------------------
### EpisodeRewardManager
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-reward.md
Documentation for the EpisodeRewardManager class, including its constructor, __call__ method, parameters, return values, and examples.
```APIDOC
## EpisodeRewardManager
### Description
Assigns the episode outcome to the last response token of each row in a `verl.protocol.DataProto` batch. This is the framework-side reward manager used when bash coding is disabled.
### Constructor
#### Parameters
- **tokenizer** (`PreTrainedTokenizer`) - Required - Used to decode prompts/responses for logging.
- **num_examine** (`int`) - Required - Max number of rows printed per `data_source` for inspection.
- **normalize_by_length** (`bool`) - Optional, default `False` - If true, `score = episode_rewards / episode_lengths` instead of raw `episode_rewards`.
### `__call__(data: DataProto, return_dict=False)`
#### Description
Python callable object used by verl's `compute_reward`. If `"rm_scores"` is already present in `data.batch`, it is returned as-is. Otherwise, it computes a sparse reward tensor with the episode score at the last valid response token of each row.
#### Parameters
- **data** (`DataProto`) - Required - The batch containing responses, attention masks, and non-tensor batch fields (`data_source`, `episode_rewards`, `episode_lengths`).
- **return_dict** (`bool`) - Optional, default `False` - If true, returns a dictionary with `reward_tensor` and `reward_extra_info`.
#### Returns
- If `return_dict=False`: `torch.FloatTensor` of shape `[batch_size, response_length]` with the episode score only at each row's last valid response token.
- If `return_dict=True`: `{"reward_tensor": tensor, "reward_extra_info": {}}`.
#### Example
```python
from agent_system.reward_manager import EpisodeRewardManager
reward_fn = EpisodeRewardManager(tokenizer=tokenizer, num_examine=1, normalize_by_length=False)
reward_tensor = reward_fn(data_batch) # [B, resp_len] sparse reward
assert float(reward_tensor[i, valid_len_i - 1]) == float(data_batch.non_tensor_batch["episode_rewards"][i])
with_dict = reward_fn(data_batch, return_dict=True) # {"reward_tensor": ..., "reward_extra_info": {}}
```
#### Handling `rm_scores` pass-through
```python
batch.batch.set_("rm_scores", rm_output)
out = reward_fn(batch, return_dict=True)
assert torch.equal(out["reward_tensor"], batch.batch["rm_scores"])
```
#### Throws
- `KeyError` - Row missing any of `attention_mask`, `responses`, `data_source`, `episode_rewards`, `episode_lengths` in the expected places.
- `AssertionError` - Shape mismatch or indexing errors from torch/numpy.
```
--------------------------------
### Refusing ambiguous configs
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-algorithm-integration.md
Demonstrates how the system refuses ambiguous configurations where multiple bash coding algorithms are enabled. The example catches a ValueError and prints the error message.
```python
try:
_get_enabled_bash_coding_algorithm({"enable_A3_algo": 1, "enable_hgpo": 1})
except ValueError as exc:
print(exc) # "Multiple bash coding algorithms are enabled: ..."
```
--------------------------------
### Direct execution with run_in_unshare_sandbox
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-sandbox-exec.md
Direct execution example: calls run_in_unshare_sandbox with a work directory, command, timeout, and extra environment variables. The result is an ExecResult object; the return code and stdout are printed.
```python
from main_entry.cli_agent_bash_coding.env_package.bash_coding.sandbox_exec import (
run_in_unshare_sandbox, ExecResult, SandboxError,
)
result = run_in_unshare_sandbox(
work_dir="/tmp/bash_coding_abc",
command="ls -la && echo done",
timeout_s=10,
extra_env={"PATH": "/usr/bin:/bin"},
)
assert isinstance(result, ExecResult)
print(result.returncode, result.stdout)
```
--------------------------------
### Configure A3 training with mixed benchmarks
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md
YAML configuration for the A3 algorithm's main settings. Set the benchmark list, base model path, batch sizes, and A3 hyperparameters. The example shows the bench list for mixed-benchmark training.
```yaml
settings:
bench: ["agentbench_os", "databench", "shellops", "ehrcon_curated", "agentbench_dbbench", "tablebench"]
```
--------------------------------
### Emulating actor dispatch for a custom algorithm
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-algorithm-integration.md
Shows how to use _get_enabled_bash_coding_algorithm and _inject_algorithm_hparams to emulate the actor dispatch for a custom algorithm. The example uses a FakeActorConfig dict-like config and asserts the algorithm name and hyperparameter injection.
```python
from main_entry.cli_agent_bash_coding.algo.registry import list_bash_coding_algorithms
from verl.workers.actor.dp_actor import _get_enabled_bash_coding_algorithm, _inject_algorithm_hparams
class FakeActorConfig(dict): # dict-like actor config
pass
cfg = FakeActorConfig({"enable_A3_algo": 1, "algorithm_w_trie": 0.65})
algo = _get_enabled_bash_coding_algorithm(cfg)
assert algo.name == "A3_algo"
kwargs = _inject_algorithm_hparams(
algo.compute_advantage, cfg, {"data": batch, "multi_turn": True, "tokenizer": tk})
assert kwargs["w_trie"] == 0.65
```
--------------------------------
### BashCodingWorker.reset(kwargs)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Resets the environment to start a new episode. Creates a temporary working directory, optionally copies an initial directory, snapshots filesystem state, and returns the initial observation and info dict.
```APIDOC
## BashCodingWorker.reset(kwargs: Dict = None) -> Tuple[str, Dict]
### Description
Resets the environment to start a new episode. Creates a temporary working directory, optionally copies an initial directory, snapshots the initial filesystem state, and prepares the reward cache. Returns the initial observation and info dict.
### Method
Method call
### Endpoint
`reset(kwargs)`
### Parameters
- **kwargs** (`Dict`, optional) - Required - Must include `"task"` (KeyError otherwise). May include `"init_dir"` for copying an initial directory.
### Behavior
- Creates temp work dir via `tempfile.mkdtemp(prefix="bash_coding_")`
- Optionally `shutil.copytree(init_dir, work_dir)` (ValueError if `init_dir` is not a directory)
- Snapshots init filesystem state and prepares reward cache
- Returns `(obs, info)` with `info["task"]`
- Observation goes through `harness.build_reset_obs` (see bash-coding-harness.md)
### Returns
- **Tuple[str, Dict]** - Observation and info dict
```
--------------------------------
### Parsing a model response into a structured action
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-action-schema.md
Use this snippet to parse a model response into a structured action, which drives the `bash_coding_projection`. The example shows how to extract the action name, plan, and code, and verify the payload span. Note that `enable_commit=False` is used here.
```python
from main_entry.cli_agent_bash_coding.action_schema import (
parse_bash_coding_action, get_bash_coding_action_spans,
)
text = 'submit_codeList filesls -la'
act = parse_bash_coding_action(text, enable_commit=False) # CodeAction(plan="List files", code="ls -la")
spans = get_bash_coding_action_spans(text, enable_commit=False)
assert text[spans.payload[0]:spans.payload[1]] == "ls -la"
```
--------------------------------
### Bash launcher invocation for bash coding with A3 algorithm
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-main-ppo.md
Abridged from launch_bash_coding.sh. Enables BASH_CODING_ENV and runs verl.trainer.main_ppo with GRPO, A3 algorithm flags, Qwen3-14B model, sglang rollout, and BashCodingMultiProcessEnv. Sets turn weight 0.7, batch size 16, max prompt length 32768, 6 env steps, n=4, 8 GPUs, and enables KL oscillation stop.
```bash
export BASH_CODING_ENABLE=1
python3 -m verl.trainer.main_ppo \
algorithm.adv_estimator=grpo \
+algorithm.enable_A3_algo=1 \
+actor_rollout_ref.actor.enable_A3_algo=1 \
+actor_rollout_ref.actor.algorithm_name=A3_algo \
+actor_rollout_ref.actor.algorithm_turn_weight=0.7 \
"data.train_files=main_entry/data/shellops/train.parquet" \
data.train_batch_size=16 data.max_prompt_length=32768 \
actor_rollout_ref.model.path=Qwen/Qwen3-14B \
actor_rollout_ref.rollout.name=sglang \
env.env_name=bash_coding/BashCodingMultiProcessEnv \
env.max_steps=6 env.rollout.n=4 \
trainer.project_name=my_run trainer.n_gpus_per_node=8 \
+trainer.kl_oscillation_stop.enable=1 etc.
```
--------------------------------
### Hydra entrypoint: main(config)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-main-ppo.md
Hydra entrypoint for the training/inference launcher. The launcher passes ~150 CLI overrides rooted at `data.*`, `actor_rollout_ref.*`, `algorithm.*`, `env.*`, `reward.*`, `trainer.*`, and `ray_init.*`.
```python
@hydra.main(config_path="config", config_name="ppo_trainer", version_base=None)
def main(config):
run_ppo(config)
```
--------------------------------
### _ValueSpan dataclass definition
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-reward.md
Internal frozen dataclass holding token spans (start, end) for response components: full_response, plan, payload, and optional commit. Not exported.
```python
@dataclass(frozen=True)
class _ValueSpan:
full_response: Tuple[int, int] | None
plan: Tuple[int, int] | None
payload: Tuple[int, int] | None
commit: Tuple[int, int] | None = None
```
--------------------------------
### main() / __main__ - CLI
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md
Command-line interface for computing semantic similarity between two texts.
```APIDOC
## CLI: python semantic_similarity.py [options]
### Description
Command-line interface for computing semantic similarity between two texts.
### Method
CLI
### Endpoint
N/A
### Parameters
#### Positional Arguments
- **text_a** (string) - Required - First text.
- **text_b** (string) - Required - Second text.
#### Optional Arguments
- **--api-base-url** (string) - Optional - Base URL (default: "http://127.0.0.1:30003").
- **--model** (string) - Optional - Model name (default: "default").
- **--timeout** (integer) - Optional - HTTP timeout in seconds (default: 60).
- **--batch-size** (integer) - Optional - Batch size (default: 32).
- **--demo** (flag) - Optional - Use demo pairs.
### Request Example
```bash
python semantic_similarity.py "text_a" "text_b" --api-base-url http://127.0.0.1:30003 --model default --timeout 60 --batch-size 32
```
### Response
#### Success Response (0)
- **score** (float) - The similarity score.
#### Response Example
```
0.8
```
### Error Handling
- Exits with non-zero code on error.
```
--------------------------------
### SearchMemory.fetch rendering format
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md
Shows the fetch method signature and the line format used for each history entry: 'Step : \n'. The obs_key and action_key parameters have no defaults and are keyword-required.
```python
def fetch(
self,
history_length: int,
obs_key: str,
action_key: str,
) -> Tuple[List[str], List[int]]:
...
lines.append(f"Step {step_num}:{act} {obs}\n")
```
--------------------------------
### Missing tokenizer raises ValueError
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-a3.md
Shows that passing tokenizer=None to compute_advantage raises a ValueError with the message starting "A3_algo requires a tokenizer to decode responses into bash ...". The ellipsis in the call hides the other arguments.
```python
try:
algo.compute_advantage(data, multi_turn=True, ..., tokenizer=None, ...)
except ValueError as exc:
print(exc) # "A3_algo requires a tokenizer to decode responses into bash ..."
```
--------------------------------
### Launch A3 training
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md
Changes to the A3 algorithm directory and runs the training launch script. Checkpoints are written to main_entry/cli_agent_bash_coding/checkpoints/ and results to main_entry/cli_agent_bash_coding/results/.
```bash
cd main_entry/cli_agent_bash_coding/train/A3_algo
bash launch_bash_coding.sh
```
--------------------------------
### Activate RetroAgent from config and use its context
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-baselines.md
Initializes the BaselineMethodManager from config, asserts RetroAgent is enabled, prepares a batch of tasks, builds prompt contexts for each task, and applies post-rollout with the manager. Requires a config with reward.retroagent_enable set and a tokenizer, rollout worker group, and freeform generation function.
```python
from main_entry.cli_agent_bash_coding.baseline_methods import BaselineMethodManager
mgr = BaselineMethodManager.from_config(cfg) # reads reward.retroagent_enable etc.
assert mgr.enabled("retroagent")
mgr.prepare_batch(tasks=task_list, is_eval=False, group_n=4)
for i in range(len(task_list)):
ctx = mgr.build_prompt_context(i)
if ctx:
print(f"row {i}: {ctx}")
stats = mgr.apply_post_rollout(
tasks=..., total_batch_list=..., total_infos=..., final_infos=...,
episode_rewards=..., group_n=4, tokenizer=tk, actor_rollout_wg=wg,
meta_info=..., generate_freeform_texts=generate,
)
```
--------------------------------
### make_envs(cfg)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md
Creates training and validation environments from a Hydra configuration object. The configuration must specify an environment name of 'bash_coding' and a rollout count that is an integer.
```APIDOC
## make_envs(cfg)
### Description
Creates a pair of training and validation environments for bash coding tasks based on the provided Hydra configuration. The configuration must specify an environment name of 'bash_coding' and a rollout count that is an integer.
### Parameters
- **cfg** (OmegaConf / DictConfig) - Required - The fully-resolved Hydra configuration object. Must contain `env.env_name` set to `"bash_coding"` and `env.rollout.n` as an integer.
### Returns
- **tuple** - A tuple `(train_envs, val_envs)` where each is an environment object supporting `reset(kwargs=...)`.
### Throws
- **ValueError** - If `config.env.env_name` does not contain `"bash_coding"` (env_manager.py:527).
- **ValueError** - If `config.env.rollout.n` is not an integer (env_manager.py:531).
- **ValueError** - If the underlying `build_bash_coding_harness` is called with an unsupported `bash_coding_harness` name (via harness/manager.py:18).
- **KeyError** - If `env_kwargs` is missing the `bash_coding_harness` key (env_manager.py:559).
### Example Usage
```python
from omegaconf import OmegaConf
from main_entry.cli_agent_bash_coding.env_manager import make_envs
cfg = OmegaConf.load("main_entry/cli_agent_bash_coding/train/A3_algo/config.yaml")
# (in practice cfg is the fully-resolved Hydra config produced by verl's entrypoint)
train_envs, val_envs = make_envs(cfg)
obs, infos = train_envs.reset(kwargs=batch_env_kwargs)
```
### Error Example
```python
import omegaconf
cfg = omegaconf.OmegaConf.create({"env": {"env_name": "babi", "rollout": {"n": 1}}})
try:
make_envs(cfg)
except ValueError as exc:
print(exc) # "cli_agent_bash_coding.make_envs only supports bash_coding env_name; got 'babi'"
```
```
--------------------------------
### Convert benchmark directory with prepare_bash_coding.py
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/data-preprocess.md
Use this bash command to convert a benchmark directory. It requires the --local_dir, --local_dataset_path, and optional --default_string_match, --fuzzy_threshold, and --num_proc flags. The --local_dataset_path must contain train.parquet/test.parquet files, otherwise a FileNotFoundError is raised.
```bash
python main_entry/cli_agent_bash_coding/data_preprocess/prepare_bash_coding.py \
--local_dir main_entry/data/shellops \
--local_dataset_path /path/to/shellops_raw \
--default_string_match fuzzy --fuzzy_threshold 0.8 --num_proc 4
```
--------------------------------
### SimpleMemory round trip with fetch
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md
Records two steps of bash-coding actions and observations, then fetches the last two steps. Asserts that both contexts have length 2 and prints the first context, which shows the formatted history lines.
```python
from agent_system.memory import SimpleMemory
mem = SimpleMemory()
mem.reset(batch_size=2)
mem.store({
"observation": ["obs A1", "obs B1"],
"observation_for_history": ["obs A1", "obs B1"],
"action": ["ls -la", "grep foo file"],
})
mem.store({
"observation": ["obs A2", "obs B2"],
"observation_for_history": ["obs A2", "obs B2"],
"action": ["cat x", "echo hi"],
})
contexts, lengths = mem.fetch(history_length=2, obs_key="observation_for_history", action_key="action")
assert lengths == [2, 2]
print(contexts[0])
# [Observation 1: 'obs A1', Action 1: 'ls -la']\n[Observation 2: 'obs A2', Action 2: 'cat x']
```
--------------------------------
### Factory usage from launcher context
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md
Factory usage from a launcher context, equivalent to TaskRunner.run. Loads a Hydra config and creates training and validation environments via make_envs, then resets the training environments.
```python
from omegaconf import OmegaConf
from main_entry.cli_agent_bash_coding.env_manager import make_envs
cfg = OmegaConf.load("main_entry/cli_agent_bash_coding/train/A3_algo/config.yaml")
# (in practice cfg is the fully-resolved Hydra config produced by verl's entrypoint)
train_envs, val_envs = make_envs(cfg)
obs, infos = train_envs.reset(kwargs=batch_env_kwargs)
```
--------------------------------
### SimpleMemory class definition and methods in main_entry/cli_agent_bash_coding/memory.py
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md
This snippet shows the SimpleMemory class from main_entry/cli_agent_bash_coding/memory.py. It includes the __init__ method initializing _data, keys, and batch_size, the update_last method for mutating the last record, and the fetch method that renders step blocks joined by blank lines. The ellipsis (...) indicates omitted method bodies as shown in the source.
```python
class SimpleMemory:
def __init__(self):
self._data: List[List[Dict[str, Any]]] | None = None
self.keys: List[str] | None = None
self.batch_size = 0
...
def update_last(self, updates: Dict[str, List[Any]]):
...
def fetch(self, history_length, obs_key="text_obs", action_key="action"):
blocks.append(f"{step_num}\n{act}\n\n{obs}")
memory_contexts.append("\n\n".join(blocks))
```
--------------------------------
### BashCodingMultiProcessEnv.__init__
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Initializes the multi-process Bash coding environment with a seed, number of environments, group size, and per-worker resources. Creates Ray actors for each worker process.
```APIDOC
## `BashCodingMultiProcessEnv.__init__`
### Description
Initializes the multi-process Bash coding environment. Creates `env_num * group_n` worker processes, each a Ray actor `BashCodingWorker`. When `is_train=False`, asserts `group_n == 1`.
### Method
`__init__`
### Parameters
- **seed** (int) - Required - Seed for worker initialization.
- **env_num** (int) - Required - Number of distinct environments.
- **group_n** (int) - Required - Number of workers per environment group.
- **resources_per_worker** (dict) - Required - Resource specification for each worker (e.g., `{"num_cpus": 0.1}`).
- **is_train** (bool) - Optional - Whether in training mode. Defaults to `True`. If `False`, `group_n` must be 1.
- **env_kwargs** - Optional - Additional keyword arguments passed to each worker.
### Notes
- `num_processes = env_num * group_n`.
- Each worker is seeded with `seed + i // group_n`.
- Raises `AssertionError` if `is_train=False` and `group_n != 1`.
```
--------------------------------
### BashCodingWorker.__init__(seed, env_kwargs)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Initializes a BashCodingWorker instance with merged configuration from environment variables and provided kwargs. Sets up sandbox execution parameters and reward coefficients.
```APIDOC
## BashCodingWorker.__init__(seed, env_kwargs)
### Description
Initializes a BashCodingWorker instance. The `env_kwargs` dictionary is merged with configuration from the `BASH_CODING_ENV_CONFIG_JSON` environment variable, with defaults winning over env var values. Sets up sandbox execution parameters and reward coefficients.
### Method
Constructor
### Endpoint
`BashCodingWorker(seed, env_kwargs)`
### Parameters
- **seed** - Required - Seed for the environment
- **env_kwargs** (`dict`) - Required - Configuration dictionary merged with environment variable config
### Configuration Keys (defaults)
| Key | Default | Meaning |
|-----|---------|---------|
| `max_steps` | `10` | Episode step budget |
| `timeout` | `10` | Per-command sandbox timeout (s) |
| `execute_commands` | `False` | Gate for shell execution |
| `max_output_chars` | `2000` | stdout+stderr truncation cap (kept tail) |
| `max_diff_chars` | `1500` | file-diff truncation cap (kept head) |
| `max_fs_diff_lines` | `50` | Diff line budget |
| `max_created_preview_lines` | `20` | Preview lines for created files |
| `unified_diff_context_lines` | `2` | `difflib.unified_diff` context |
| `exec_backend` | `"sandbox"` | Execution backend name |
| `exec_error_penalty` | `0.2` | Negative reward on errors (`-exec_error_penalty`) |
| `answer_reward` | `1.0` | Terminal answer reward scale |
| `progress_gain_coef` | `0.3` | Progress (fs-coverage) reward scale |
| `use_model_evidence_gain` | `0` (bool(int)) | Gate semantic-similarity answer bonus |
| `no_progress_on_answer` | `True` | Suppress progress gain on answer steps |
| `require_answer_for_done` | `True` | Episode only ends via `submit_answer` |
```
--------------------------------
### BashCodingMultiProcessEnv.step
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Steps the environment with the given actions. Asserts the number of actions matches the active number of processes.
```APIDOC
## `BashCodingMultiProcessEnv.step`
### Description
Executes a step for each active process using the provided actions. Asserts that `len(actions) == active_n`.
### Method
`step`
### Parameters
- **actions** (list) - Required - List of actions to apply to each active process.
### Returns
- **tuple** - `(obs_list, reward_list, done_list, info_list)`.
```
--------------------------------
### Build and use multiprocess batch environment with build_bash_coding_envs
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Builds a BashCodingMultiProcessEnv with 4 environments and 2 groups per environment, using 0.1 CPU per worker. Resets with a list of task and reward_spec kwargs, then closes the environment.
```python
from main_entry.cli_agent_bash_coding.env_package.bash_coding import build_bash_coding_envs
envs = build_bash_coding_envs(
seed=0, env_num=4, group_n=2, resources_per_worker={"num_cpus": 0.1}, is_train=True,
max_steps=6, timeout=10, exec_backend="sandbox", bash_coding_harness="vanilla",
)
obs_list, info_list = envs.reset(kwargs=[{"task": t, "reward_spec": spec} for t, spec in zip(tasks, specs)])
...
envs.close()
```
--------------------------------
### Algorithm registry and lookup functions (algo/registry.py)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-gigpo-hgpo.md
Defines the _ALGORITHMS dictionary with entries for 'A3_algo' and 'hgpo', plus functions to retrieve and list algorithms. The lookup is case-insensitive and returns None for empty or unknown names.
```python
_ALGORITHMS = {
"A3_algo": BashCodingAlgorithm(
name="A3_algo",
compute_advantage=compute_A3_algo_advantage,
compute_policy_loss=compute_policy_loss_A3_algo,
),
"hgpo": BashCodingAlgorithm(
name="hgpo",
),
}
def get_bash_coding_algorithm(name: str) -> BashCodingAlgorithm | None:
key = str(name).strip().lower()
if key == "":
return None
for reg_key, algo in _ALGORITHMS.items():
if str(reg_key).lower() == key:
return algo
return None
def list_bash_coding_algorithms() -> tuple[str, ...]:
return tuple(_ALGORITHMS.keys())
```
--------------------------------
### Configure inference checkpoint pair
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md
YAML configuration for inference. Fill model.checkpoint_pair with the trained checkpoint and base model path. Once set, the launcher enters TEST MODE automatically. Use vanilla for standard harness and sigma_reveal_rd for sigma-Reveal.
```yaml
model:
checkpoint_pair: ":"
pretrained_model_path_default: "Qwen/Qwen3-14B"
```
--------------------------------
### BashIntentActionSpace.__init__
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md
Initializes a BashIntentActionSpace instance with an optional tree-sitter parser. If no parser is provided, a default parser for bash is created.
```APIDOC
## BashIntentActionSpace.__init__
### Description
Initializes a BashIntentActionSpace instance. Accepts an optional tree-sitter Parser; if none is given, a default parser for the bash language is created.
### Method
Constructor
### Signature
```python
__init__(self, parser: Parser | None = None) -> None
```
### Parameters
- **parser** (Parser | None) - Optional. A tree-sitter Parser instance. If None, a default parser for bash is used.
### Notes
Requires `tree-sitter` and `tree_sitter_bash` packages.
```
--------------------------------
### Build sigma_reveal_rd harness
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-harness.md
Constructs the sigma_reveal_rd harness and verifies its base flags. The build_text_obs call renders the task prompt template around the provided observation.
```python
from main_entry.cli_agent_bash_coding.harness import build_bash_coding_harness
harness = build_bash_coding_harness({"bash_coding_harness": "sigma_reveal_rd"})
assert harness.name == "sigma_reveal_rd"
assert harness.sigma_reject_enabled is False # base flags
obs = harness.build_text_obs(env_manager, ["first obs"], init=True)
# obs[0] contains the task prompt template rendered around "first obs"
```
--------------------------------
### Intent-distance clustering with BashIntentActionSpace
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md
Creates a BashIntentActionSpace instance and computes a distance matrix for a list of shell commands. Also prints the intent signature for a sample command. Useful for clustering similar bash intents.
```python
from main_entry.cli_agent_bash_coding.tooling.action_space_similarity import BashIntentActionSpace
space = BashIntentActionSpace()
shells = ["ls | wc -l", "ls *.txt | wc -l", "echo $HOME", "grep error log.txt"]
d = space.distance_matrix(shells)
print(space.intent_signature("ls | wc -l")) # ['K:pipeline', 'V:ls', 'K:list', 'V:wc', 'W:wc', 'W:l', ...]
```
--------------------------------
### BashCodingTrajectoryCollector class definition and initialization
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-rollout.md
Defines the class with history delimiters and initializes the harness, evidence gain settings, and validation dumps. The harness is built from config.env, and evidence gain coefficient is set if enabled.
```python
class BashCodingTrajectoryCollector(TrajectoryCollector):
HISTORY_PREFIX = "Here is the history records:\n"
HISTORY_SUFFIX = "\n\nYou are now at step "
def __init__(self, config, tokenizer, processor=None):
super().__init__(config=config, tokenizer=tokenizer, processor=processor)
self.harness = build_bash_coding_harness(
OmegaConf.to_container(config.env, resolve=True)
)
self.enable_evidence_gain = bool(config.env.use_model_evidence_gain)
self._validation_dataset_dump: list[str] = []
self._validation_prompt_dump: list[str] = []
if self.enable_evidence_gain:
self.evidence_gain_coef = float(config.env.progress_gain_coef)
else:
self.evidence_gain_coef = None
```
--------------------------------
### make_envs(config) - build train and val bash coding environments
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md
Builds training and validation environments for bash coding tasks. Raises ValueError if env_name lacks 'bash_coding' or if env.rollout.n is not an int. Train envs use group_n = n (or 1 if n <= 0) and is_train=True; val envs use group_n=1, is_train=False, and seed+1000. Both share the same projection_f and are wrapped in BashCodingEnvironmentManager.
```python
def make_envs(config):
if "bash_coding" not in config.env.env_name.lower():
raise ValueError(...)
if not isinstance(config.env.rollout.n, int):
raise ValueError(...)
group_n = config.env.rollout.n if config.env.rollout.n > 0 else 1
...
train_envs = build_bash_coding_envs(seed=config.env.seed, env_num=config.data.train_batch_size,
group_n=group_n, resources_per_worker=..., is_train=True, **env_kwargs)
val_envs = build_bash_coding_envs(seed=config.env.seed + 1000, env_num=config.data.val_batch_size,
group_n=1, resources_per_worker=..., is_train=False, **env_kwargs)
projection_f = partial(bash_coding_projection,
enable_commit=uses_commit_action_schema(env_kwargs["bash_coding_harness"]))
envs = BashCodingEnvironmentManager(train_envs, projection_f, config)
val_envs = BashCodingEnvironmentManager(val_envs, projection_f, config)
return envs, val_envs
```
--------------------------------
### BashCodingEnvironmentManager __init__ method
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md
Initializes the environment manager with memory selection based on BASH_CODING_ENABLE, method manager, harness, and sigma-related fields. Reads environment variables for semantic similarity URL and batch size.
```python
class BashCodingEnvironmentManager(EnvironmentManagerBase):
def __init__(self, envs, projection_f, config):
enabled = str(os.environ.get("BASH_CODING_ENABLE", "0")).strip() == "1"
self.memory = ResearchSimpleMemory() if enabled else BaselineSimpleMemory()
self.method_manager = BaselineMethodManager.from_config(config)
self.harness = build_bash_coding_harness(
OmegaConf.to_container(config.env, resolve=True)
)
self.sample_kwargs_list: List[Dict[str, Any]] = []
self.semantic_similarity_url = os.environ.get(
"BASH_CODING_SEMANTIC_SIMILARITY_URL", "http://127.0.0.1:30003"
)
self.semantic_similarity_batch_size = int(
os.environ.get("BASH_CODING_SEMANTIC_SIMILARITY_BATCH_SIZE", "32")
)
self._sigma_space: BashIntentActionSpace | None = (
BashIntentActionSpace() if else None
)
self._sigma_seen: List[Set[Tuple[str, ...]]] = []
self._sigma_counts: List[Counter] = []
self._sigma_concentration_tau: float = float(
getattr(config.env, "sigma_concentration_tau", 0.8)
)
super().__init__(envs, projection_f, config)
```
--------------------------------
### Repository directory tree
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/README.md
Shows the top-level directory structure and key files for the agent_system, main_entry/cli_agent_bash_coding, verl, main_entry/ehrcon_rebuild, environ, and assets directories. Use this to navigate the codebase and locate the framework-agnostic RL agent base, the A3 domain code, and the vendored verl integration points.
```text
agent_system/ # framework-agnostic multi-turn RL agent base
environments/ base.py EnvironmentManagerBase, to_numpy
__init__.py re-exports EnvironmentManagerBase + make_envs
memory/ base.py BaseMemory (ABC)
memory.py SimpleMemory, SearchMemory
multi_turn_rollout/ rollout_loop.py TrajectoryCollector
utils.py to_list_of_dict, torch_to_numpy, numpy_to_torch,
process_image, adjust_batch, filter_group_data
reward_manager/ episode.py EpisodeRewardManager
main_entry/cli_agent_bash_coding/ # A3 domain code (the novel part)
algo/ base.py BashCodingAlgorithm (dataclass)
registry.py get_bash_coding_algorithm, list_bash_coding_algorithms
A3_algo.py compute_A3_algo_advantage, compute_policy_loss_A3_algo,
compute_policy_loss_gspo_sequence
stable_ops.py normalization/fusion/trust-region helpers
hgpo.py compute_hgpo_advantage (HGPO baseline)
core_gigpo.py compute_gigpo_outcome_advantage (GiGPO baseline)
env_manager.py BashCodingEnvironmentManager, make_envs
env_package/bash_coding/ BashCodingWorker, BashCodingMultiProcessEnv (ray),
build_bash_coding_envs, bash_coding_projection,
sandbox_exec (run_in_unshare_sandbox, ExecResult, SandboxError)
harness/ base.py BashCodingHarnessBase
enhanced.py EnhancedBashCodingHarness (FILE_CHANGES)
vanilla.py VanillaBashCodingHarness
sigma_reveal_rd.py SigmaRevealRDBashCodingHarness (sigma-Reveal)
manager.py build_bash_coding_harness (registry)
memory.py / action_schema.py / cib_action_encoding.py / reward_provenance.py
reward/ segmented_episode_reward.py SegmentedEpisodeRewardManager
rstar.py apply_rstar_resampling
rollout/ evidence_collector.py BashCodingTrajectoryCollector
prompts/ bash_coding.py BASH_CODING_TEMPLATE, BASH_CODING_TEMPLATE_NO_HIS
baseline_methods/ BaselineMethodManager, CommitCreditRuntime, RetroAgentRuntime,
lats_infer.lats_step (LATS inference baseline)
tooling/ action_space_similarity/bash_intent_action_space.py BashIntentActionSpace
semantic_similarity/semantic_similarity.py OpenAI-compatible LLM judger client
data_preprocess/ prepare_bash_coding.py (CLI), multibench_schema.py (normalize_example)
eval/ eval_results_jsonl.py (CLI + lib), merge_results.py, align_results_to_test_parquet_rerun_eval.py
infer/ pure_infer.py (CLI: API-server inference against a local SGLang server)
train/ config.yaml variants, launch_bash_coding.sh launchers, load_config_yaml.py
tooling/ add_dataset_field_to_parquet.py
verl/ # vendored verl framework + A3 integration points
protocol.py DataProto
trainer/main_ppo.py main() Hydra entrypoint, CoupledReward, run_ppo, TaskRunner
trainer/ppo/ray_trainer.py RayPPOTrainer (rstar hook, kl_oscillation_stop, HGPO adv)
trainer/ppo/core_algos.py compute_grpo_outcome_advantage, compute_policy_loss, ...
workers/actor/dp_actor.py DataParallelPPOActor (A3 dispatch)
workers/fsdp_workers.py FSDP actor (A3 advantage dispatch)
utils/torch_functional.py tokenize_and_postprocess_data, masked_whiten, ...
main_entry/ehrcon_rebuild/ EHRCon benchmark reconstruction scripts
environ/ conda env + apt bootstrap scripts
assets/ paper figures
```
--------------------------------
### Generate Freeform Texts for Reflection Baselines
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-rollout.md
Generates freeform texts for reflection baselines using the _generate_freeform_texts method. Requires an actor_rollout_wg and meta_info from a gen_batch. Uses max_prompt_tokens=12000, hard_truncate_tokens=16000, and data_source='retroagent_reflection'.
```python
texts = collector._generate_freeform_texts(
prompts=["Reflect on the last episode..."],
actor_rollout_wg=actor_rollout_wg,
meta_info=gen_batch.meta_info,
max_prompt_tokens=12000, hard_truncate_tokens=16000, data_source="retroagent_reflection",
)
```
--------------------------------
### BashCodingWorker.step(action)
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md
Executes a single action in the environment. Handles empty commands, format violations, answer submissions, and code execution with full reward and error handling.
```APIDOC
## BashCodingWorker.step(action: str) -> Tuple[str, float, bool, Dict]
### Description
Executes a single action in the environment. Handles empty commands, format violations, answer submissions, and code execution with full reward and error handling.
### Method
Method call
### Endpoint
`step(action)`
### Parameters
- **action** (`str`) - Required - The action string to execute
### Behavior
| Input | Behavior |
|-------|----------|
| empty action | `("Error: Empty command is not allowed.", reward, done, info)` — `done = step_count >= max_steps`; if final step and `require_answer_for_done`, terminal-submission provenance (0) is added and failed-terminal progress is zeroed |
| `FORMAT_VIOLATION_PREFIX` | `("Error: Format violation.", ...)` same structure |
| `ANSWER_PREFIX` | `done=True`; reward from `_calculate_reward`; `won = is_correct = _check_task_completion(answer_text)`; info has `direct_answer=True`; incorrect answers zero the terminal progress |
| anything else | `_execute_action(action)` → sandbox run; success/progress rewards; `done = step_count >= max_steps` (when `require_answer_for_done`) with final-step bookkeeping; `SandboxError` → `SandboxError: ` observation, `-exec_error_penalty`, `done=False`; other `Exception` → `EnvError: : ` observation, `-exec_error_penalty`, `done=True` |
### Info Dict Fields
Every `info` dict contains at least: `won`, `error`, `returncode`, `stdout`, `stderr`, `work_dir`, `changed_files`, `diff`, `step`, `env_warnings`, `reward_provenance`, `reward_total`, plus harness extras via `harness.build_step_info(self)` and (exec path) `obs_truncation`.
### Returns
- **Tuple[str, float, bool, Dict]** - Observation, reward, done flag, and info dict
```
--------------------------------
### Training data flow
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/README.md
Describes the training data flow from the verl trainer entrypoint through the environment manager, trajectory collector, reward manager, and advantage computation. Use this to understand how prompts, actions, sandbox execution, and rewards are orchestrated in the A3 training loop.
```text
verl.trainer.main_ppo → TaskRunner.run → make_envs(config) builds ray BashCodingMultiProcessEnv workers wrapped by BashCodingEnvironmentManager; BashCodingTrajectoryCollector.multi_turn_loop drives prompt→action→sandbox-execute→observation rounds; per-step rewards and reward_provenance are accumulated into episode_rewards; SegmentedEpisodeRewardManager distributes the episode score over response token spans; the actor worker calls compute_A3_algo_advantage (dispatched by _get_enabled_bash_coding_algorithm, hyperparameters injected from actor.algorithm_* config keys) then compute_policy_loss_A3_algo.
```
--------------------------------
### EnvironmentManagerBase class definition
Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-environments.md
Base class for multi-turn environment wrappers. Constructor stores envs, projection function, and config. Includes reset, step, build_text_obs, close, success_evaluator, _process_batch, and save_image methods.
```python
class EnvironmentManagerBase:
def __init__(self, envs, projection_f, config):
self.envs = envs
self.projection_f = projection_f
self.config = config
def reset(self, kwargs) -> Dict[str, Any]:
obs, infos = self.envs.reset()
return {"text": None, "image": obs, "anchor": None}, infos
def step(self, text_actions: List[str]):
actions, valids = self.projection_f(text_actions)
next_obs, rewards, dones, infos = self.envs.step(actions)
next_observations = {"text": None, "image": next_obs, "anchor": None}
for i, info in enumerate(infos):
info["is_action_valid"] = to_numpy(valids[i])
rewards = to_numpy(rewards)
dones = to_numpy(dones)
return next_observations, rewards, dones, infos
def build_text_obs(self) -> List[str]:
pass
def close(self) -> None:
self.envs.close()
def success_evaluator(self, *args, **kwargs) -> Dict[str, np.ndarray]:
total_infos = kwargs["total_infos"]
total_batch_list = kwargs["total_batch_list"]
batch_size = len(total_batch_list)
success = defaultdict(list)
for bs in range(batch_size):
self._process_batch(bs, total_batch_list, total_infos, success)
assert len(success["success_rate"]) == batch_size
return {key: np.array(value) for key, value in success.items()}
def _process_batch(self, batch_idx, total_batch_list, total_infos, success):
for i in reversed(range(len(total_batch_list[batch_idx]))):
batch_item = total_batch_list[batch_idx][i]
if batch_item["active_masks"]:
info = total_infos[batch_idx][i]
won_value = float(info["won"])
success["success_rate"].append(won_value)
return
def save_image(self, image, step):
path = os.path.join(
os.path.dirname(__file__), os.path.join("images", self.config.env.env_name)
)
if not os.path.exists(path):
os.makedirs(path)
path = os.path.join(path, f"step{step}.png")
if isinstance(image, torch.Tensor):
image = image.detach().cpu().numpy()
if isinstance(image, np.ndarray):
pass
else:
raise ValueError(f"Unsupported type: {type(image)})")
if len(image.shape) == 4:
image = image[0]
if image.shape[0] == 3:
image = np.transpose(image, (1, 2, 0))
if image.max() <= 1.0:
image = image * 255
image = image.astype(np.uint8)
from PIL import Image
image = Image.fromarray(image)
image.save(path)
```