### Registry lookup and compute_advantage example Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-a3.md Demonstrates how verl's actor retrieves the A3_algo algorithm from the registry and calls compute_advantage. Requires a tokenizer; missing tokenizer raises an error (see next example). The advantages are stored at data.batch["advantages"], and debug metrics include actor/A3_algo_tree_gate_mean. ```python from main_entry.cli_agent_bash_coding.algo.registry import ( get_bash_coding_algorithm, list_bash_coding_algorithms, ) assert "A3_algo" in list_bash_coding_algorithms() algo = get_bash_coding_algorithm("A3_algo") assert algo.name == "A3_algo" data = algo.compute_advantage( data=batch, multi_turn=True, epsilon=1e-6, adv_bound=1.1, trie_min_count=2, w_trie=0.65, support_prior=0.1, decision_gamma=0.9, w_action_cluster=1.0, tokenizer=tokenizer, intent_scopes=[1, 2, 3, -1], w_intent_scopes=[0.25, 0.25, 0.25, 0.25], A3_algo_tree_intent_match_tau=0.05, ) adv = data.batch["advantages"] # [B, resp_len] token-level advantages dbg = data.meta_info["algorithm_debug_metrics"] print(dbg["actor/A3_algo_tree_gate_mean"]) ``` -------------------------------- ### Create conda environment and install CLI dependencies Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md Creates the conda environment from environment.yml, activates it, and runs install_apt.sh to prepare apt libraries required for CLI-agent execution in the sandbox. The research implementation is docker-free. ```bash conda env create -f environ/environment.yml conda activate cli_agent bash main_entry/pre_build/install_apt.sh ``` -------------------------------- ### EpisodeRewardManager Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-reward.md Documentation for the EpisodeRewardManager class, including its constructor, __call__ method, parameters, return values, and examples. ```APIDOC ## EpisodeRewardManager ### Description Assigns the episode outcome to the last response token of each row in a `verl.protocol.DataProto` batch. This is the framework-side reward manager used when bash coding is disabled. ### Constructor #### Parameters - **tokenizer** (`PreTrainedTokenizer`) - Required - Used to decode prompts/responses for logging. - **num_examine** (`int`) - Required - Max number of rows printed per `data_source` for inspection. - **normalize_by_length** (`bool`) - Optional, default `False` - If true, `score = episode_rewards / episode_lengths` instead of raw `episode_rewards`. ### `__call__(data: DataProto, return_dict=False)` #### Description Python callable object used by verl's `compute_reward`. If `"rm_scores"` is already present in `data.batch`, it is returned as-is. Otherwise, it computes a sparse reward tensor with the episode score at the last valid response token of each row. #### Parameters - **data** (`DataProto`) - Required - The batch containing responses, attention masks, and non-tensor batch fields (`data_source`, `episode_rewards`, `episode_lengths`). - **return_dict** (`bool`) - Optional, default `False` - If true, returns a dictionary with `reward_tensor` and `reward_extra_info`. #### Returns - If `return_dict=False`: `torch.FloatTensor` of shape `[batch_size, response_length]` with the episode score only at each row's last valid response token. - If `return_dict=True`: `{"reward_tensor": tensor, "reward_extra_info": {}}`. #### Example ```python from agent_system.reward_manager import EpisodeRewardManager reward_fn = EpisodeRewardManager(tokenizer=tokenizer, num_examine=1, normalize_by_length=False) reward_tensor = reward_fn(data_batch) # [B, resp_len] sparse reward assert float(reward_tensor[i, valid_len_i - 1]) == float(data_batch.non_tensor_batch["episode_rewards"][i]) with_dict = reward_fn(data_batch, return_dict=True) # {"reward_tensor": ..., "reward_extra_info": {}} ``` #### Handling `rm_scores` pass-through ```python batch.batch.set_("rm_scores", rm_output) out = reward_fn(batch, return_dict=True) assert torch.equal(out["reward_tensor"], batch.batch["rm_scores"]) ``` #### Throws - `KeyError` - Row missing any of `attention_mask`, `responses`, `data_source`, `episode_rewards`, `episode_lengths` in the expected places. - `AssertionError` - Shape mismatch or indexing errors from torch/numpy. ``` -------------------------------- ### Refusing ambiguous configs Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-algorithm-integration.md Demonstrates how the system refuses ambiguous configurations where multiple bash coding algorithms are enabled. The example catches a ValueError and prints the error message. ```python try: _get_enabled_bash_coding_algorithm({"enable_A3_algo": 1, "enable_hgpo": 1}) except ValueError as exc: print(exc) # "Multiple bash coding algorithms are enabled: ..." ``` -------------------------------- ### Direct execution with run_in_unshare_sandbox Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-sandbox-exec.md Direct execution example: calls run_in_unshare_sandbox with a work directory, command, timeout, and extra environment variables. The result is an ExecResult object; the return code and stdout are printed. ```python from main_entry.cli_agent_bash_coding.env_package.bash_coding.sandbox_exec import ( run_in_unshare_sandbox, ExecResult, SandboxError, ) result = run_in_unshare_sandbox( work_dir="/tmp/bash_coding_abc", command="ls -la && echo done", timeout_s=10, extra_env={"PATH": "/usr/bin:/bin"}, ) assert isinstance(result, ExecResult) print(result.returncode, result.stdout) ``` -------------------------------- ### Configure A3 training with mixed benchmarks Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md YAML configuration for the A3 algorithm's main settings. Set the benchmark list, base model path, batch sizes, and A3 hyperparameters. The example shows the bench list for mixed-benchmark training. ```yaml settings: bench: ["agentbench_os", "databench", "shellops", "ehrcon_curated", "agentbench_dbbench", "tablebench"] ``` -------------------------------- ### Emulating actor dispatch for a custom algorithm Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-algorithm-integration.md Shows how to use _get_enabled_bash_coding_algorithm and _inject_algorithm_hparams to emulate the actor dispatch for a custom algorithm. The example uses a FakeActorConfig dict-like config and asserts the algorithm name and hyperparameter injection. ```python from main_entry.cli_agent_bash_coding.algo.registry import list_bash_coding_algorithms from verl.workers.actor.dp_actor import _get_enabled_bash_coding_algorithm, _inject_algorithm_hparams class FakeActorConfig(dict): # dict-like actor config pass cfg = FakeActorConfig({"enable_A3_algo": 1, "algorithm_w_trie": 0.65}) algo = _get_enabled_bash_coding_algorithm(cfg) assert algo.name == "A3_algo" kwargs = _inject_algorithm_hparams( algo.compute_advantage, cfg, {"data": batch, "multi_turn": True, "tokenizer": tk}) assert kwargs["w_trie"] == 0.65 ``` -------------------------------- ### BashCodingWorker.reset(kwargs) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Resets the environment to start a new episode. Creates a temporary working directory, optionally copies an initial directory, snapshots filesystem state, and returns the initial observation and info dict. ```APIDOC ## BashCodingWorker.reset(kwargs: Dict = None) -> Tuple[str, Dict] ### Description Resets the environment to start a new episode. Creates a temporary working directory, optionally copies an initial directory, snapshots the initial filesystem state, and prepares the reward cache. Returns the initial observation and info dict. ### Method Method call ### Endpoint `reset(kwargs)` ### Parameters - **kwargs** (`Dict`, optional) - Required - Must include `"task"` (KeyError otherwise). May include `"init_dir"` for copying an initial directory. ### Behavior - Creates temp work dir via `tempfile.mkdtemp(prefix="bash_coding_")` - Optionally `shutil.copytree(init_dir, work_dir)` (ValueError if `init_dir` is not a directory) - Snapshots init filesystem state and prepares reward cache - Returns `(obs, info)` with `info["task"]` - Observation goes through `harness.build_reset_obs` (see bash-coding-harness.md) ### Returns - **Tuple[str, Dict]** - Observation and info dict ``` -------------------------------- ### Parsing a model response into a structured action Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-action-schema.md Use this snippet to parse a model response into a structured action, which drives the `bash_coding_projection`. The example shows how to extract the action name, plan, and code, and verify the payload span. Note that `enable_commit=False` is used here. ```python from main_entry.cli_agent_bash_coding.action_schema import ( parse_bash_coding_action, get_bash_coding_action_spans, ) text = 'submit_codeList filesls -la' act = parse_bash_coding_action(text, enable_commit=False) # CodeAction(plan="List files", code="ls -la") spans = get_bash_coding_action_spans(text, enable_commit=False) assert text[spans.payload[0]:spans.payload[1]] == "ls -la" ``` -------------------------------- ### Bash launcher invocation for bash coding with A3 algorithm Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-main-ppo.md Abridged from launch_bash_coding.sh. Enables BASH_CODING_ENV and runs verl.trainer.main_ppo with GRPO, A3 algorithm flags, Qwen3-14B model, sglang rollout, and BashCodingMultiProcessEnv. Sets turn weight 0.7, batch size 16, max prompt length 32768, 6 env steps, n=4, 8 GPUs, and enables KL oscillation stop. ```bash export BASH_CODING_ENABLE=1 python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ +algorithm.enable_A3_algo=1 \ +actor_rollout_ref.actor.enable_A3_algo=1 \ +actor_rollout_ref.actor.algorithm_name=A3_algo \ +actor_rollout_ref.actor.algorithm_turn_weight=0.7 \ "data.train_files=main_entry/data/shellops/train.parquet" \ data.train_batch_size=16 data.max_prompt_length=32768 \ actor_rollout_ref.model.path=Qwen/Qwen3-14B \ actor_rollout_ref.rollout.name=sglang \ env.env_name=bash_coding/BashCodingMultiProcessEnv \ env.max_steps=6 env.rollout.n=4 \ trainer.project_name=my_run trainer.n_gpus_per_node=8 \ +trainer.kl_oscillation_stop.enable=1 etc. ``` -------------------------------- ### Hydra entrypoint: main(config) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/verl-main-ppo.md Hydra entrypoint for the training/inference launcher. The launcher passes ~150 CLI overrides rooted at `data.*`, `actor_rollout_ref.*`, `algorithm.*`, `env.*`, `reward.*`, `trainer.*`, and `ray_init.*`. ```python @hydra.main(config_path="config", config_name="ppo_trainer", version_base=None) def main(config): run_ppo(config) ``` -------------------------------- ### _ValueSpan dataclass definition Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-reward.md Internal frozen dataclass holding token spans (start, end) for response components: full_response, plan, payload, and optional commit. Not exported. ```python @dataclass(frozen=True) class _ValueSpan: full_response: Tuple[int, int] | None plan: Tuple[int, int] | None payload: Tuple[int, int] | None commit: Tuple[int, int] | None = None ``` -------------------------------- ### main() / __main__ - CLI Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md Command-line interface for computing semantic similarity between two texts. ```APIDOC ## CLI: python semantic_similarity.py [options] ### Description Command-line interface for computing semantic similarity between two texts. ### Method CLI ### Endpoint N/A ### Parameters #### Positional Arguments - **text_a** (string) - Required - First text. - **text_b** (string) - Required - Second text. #### Optional Arguments - **--api-base-url** (string) - Optional - Base URL (default: "http://127.0.0.1:30003"). - **--model** (string) - Optional - Model name (default: "default"). - **--timeout** (integer) - Optional - HTTP timeout in seconds (default: 60). - **--batch-size** (integer) - Optional - Batch size (default: 32). - **--demo** (flag) - Optional - Use demo pairs. ### Request Example ```bash python semantic_similarity.py "text_a" "text_b" --api-base-url http://127.0.0.1:30003 --model default --timeout 60 --batch-size 32 ``` ### Response #### Success Response (0) - **score** (float) - The similarity score. #### Response Example ``` 0.8 ``` ### Error Handling - Exits with non-zero code on error. ``` -------------------------------- ### SearchMemory.fetch rendering format Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md Shows the fetch method signature and the line format used for each history entry: 'Step : \n'. The obs_key and action_key parameters have no defaults and are keyword-required. ```python def fetch( self, history_length: int, obs_key: str, action_key: str, ) -> Tuple[List[str], List[int]]: ... lines.append(f"Step {step_num}:{act} {obs}\n") ``` -------------------------------- ### Missing tokenizer raises ValueError Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-a3.md Shows that passing tokenizer=None to compute_advantage raises a ValueError with the message starting "A3_algo requires a tokenizer to decode responses into bash ...". The ellipsis in the call hides the other arguments. ```python try: algo.compute_advantage(data, multi_turn=True, ..., tokenizer=None, ...) except ValueError as exc: print(exc) # "A3_algo requires a tokenizer to decode responses into bash ..." ``` -------------------------------- ### Launch A3 training Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md Changes to the A3 algorithm directory and runs the training launch script. Checkpoints are written to main_entry/cli_agent_bash_coding/checkpoints/ and results to main_entry/cli_agent_bash_coding/results/. ```bash cd main_entry/cli_agent_bash_coding/train/A3_algo bash launch_bash_coding.sh ``` -------------------------------- ### Activate RetroAgent from config and use its context Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-baselines.md Initializes the BaselineMethodManager from config, asserts RetroAgent is enabled, prepares a batch of tasks, builds prompt contexts for each task, and applies post-rollout with the manager. Requires a config with reward.retroagent_enable set and a tokenizer, rollout worker group, and freeform generation function. ```python from main_entry.cli_agent_bash_coding.baseline_methods import BaselineMethodManager mgr = BaselineMethodManager.from_config(cfg) # reads reward.retroagent_enable etc. assert mgr.enabled("retroagent") mgr.prepare_batch(tasks=task_list, is_eval=False, group_n=4) for i in range(len(task_list)): ctx = mgr.build_prompt_context(i) if ctx: print(f"row {i}: {ctx}") stats = mgr.apply_post_rollout( tasks=..., total_batch_list=..., total_infos=..., final_infos=..., episode_rewards=..., group_n=4, tokenizer=tk, actor_rollout_wg=wg, meta_info=..., generate_freeform_texts=generate, ) ``` -------------------------------- ### make_envs(cfg) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md Creates training and validation environments from a Hydra configuration object. The configuration must specify an environment name of 'bash_coding' and a rollout count that is an integer. ```APIDOC ## make_envs(cfg) ### Description Creates a pair of training and validation environments for bash coding tasks based on the provided Hydra configuration. The configuration must specify an environment name of 'bash_coding' and a rollout count that is an integer. ### Parameters - **cfg** (OmegaConf / DictConfig) - Required - The fully-resolved Hydra configuration object. Must contain `env.env_name` set to `"bash_coding"` and `env.rollout.n` as an integer. ### Returns - **tuple** - A tuple `(train_envs, val_envs)` where each is an environment object supporting `reset(kwargs=...)`. ### Throws - **ValueError** - If `config.env.env_name` does not contain `"bash_coding"` (env_manager.py:527). - **ValueError** - If `config.env.rollout.n` is not an integer (env_manager.py:531). - **ValueError** - If the underlying `build_bash_coding_harness` is called with an unsupported `bash_coding_harness` name (via harness/manager.py:18). - **KeyError** - If `env_kwargs` is missing the `bash_coding_harness` key (env_manager.py:559). ### Example Usage ```python from omegaconf import OmegaConf from main_entry.cli_agent_bash_coding.env_manager import make_envs cfg = OmegaConf.load("main_entry/cli_agent_bash_coding/train/A3_algo/config.yaml") # (in practice cfg is the fully-resolved Hydra config produced by verl's entrypoint) train_envs, val_envs = make_envs(cfg) obs, infos = train_envs.reset(kwargs=batch_env_kwargs) ``` ### Error Example ```python import omegaconf cfg = omegaconf.OmegaConf.create({"env": {"env_name": "babi", "rollout": {"n": 1}}}) try: make_envs(cfg) except ValueError as exc: print(exc) # "cli_agent_bash_coding.make_envs only supports bash_coding env_name; got 'babi'" ``` ``` -------------------------------- ### Convert benchmark directory with prepare_bash_coding.py Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/data-preprocess.md Use this bash command to convert a benchmark directory. It requires the --local_dir, --local_dataset_path, and optional --default_string_match, --fuzzy_threshold, and --num_proc flags. The --local_dataset_path must contain train.parquet/test.parquet files, otherwise a FileNotFoundError is raised. ```bash python main_entry/cli_agent_bash_coding/data_preprocess/prepare_bash_coding.py \ --local_dir main_entry/data/shellops \ --local_dataset_path /path/to/shellops_raw \ --default_string_match fuzzy --fuzzy_threshold 0.8 --num_proc 4 ``` -------------------------------- ### SimpleMemory round trip with fetch Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md Records two steps of bash-coding actions and observations, then fetches the last two steps. Asserts that both contexts have length 2 and prints the first context, which shows the formatted history lines. ```python from agent_system.memory import SimpleMemory mem = SimpleMemory() mem.reset(batch_size=2) mem.store({ "observation": ["obs A1", "obs B1"], "observation_for_history": ["obs A1", "obs B1"], "action": ["ls -la", "grep foo file"], }) mem.store({ "observation": ["obs A2", "obs B2"], "observation_for_history": ["obs A2", "obs B2"], "action": ["cat x", "echo hi"], }) contexts, lengths = mem.fetch(history_length=2, obs_key="observation_for_history", action_key="action") assert lengths == [2, 2] print(contexts[0]) # [Observation 1: 'obs A1', Action 1: 'ls -la']\n[Observation 2: 'obs A2', Action 2: 'cat x'] ``` -------------------------------- ### Factory usage from launcher context Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md Factory usage from a launcher context, equivalent to TaskRunner.run. Loads a Hydra config and creates training and validation environments via make_envs, then resets the training environments. ```python from omegaconf import OmegaConf from main_entry.cli_agent_bash_coding.env_manager import make_envs cfg = OmegaConf.load("main_entry/cli_agent_bash_coding/train/A3_algo/config.yaml") # (in practice cfg is the fully-resolved Hydra config produced by verl's entrypoint) train_envs, val_envs = make_envs(cfg) obs, infos = train_envs.reset(kwargs=batch_env_kwargs) ``` -------------------------------- ### SimpleMemory class definition and methods in main_entry/cli_agent_bash_coding/memory.py Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-memory.md This snippet shows the SimpleMemory class from main_entry/cli_agent_bash_coding/memory.py. It includes the __init__ method initializing _data, keys, and batch_size, the update_last method for mutating the last record, and the fetch method that renders step blocks joined by blank lines. The ellipsis (...) indicates omitted method bodies as shown in the source. ```python class SimpleMemory: def __init__(self): self._data: List[List[Dict[str, Any]]] | None = None self.keys: List[str] | None = None self.batch_size = 0 ... def update_last(self, updates: Dict[str, List[Any]]): ... def fetch(self, history_length, obs_key="text_obs", action_key="action"): blocks.append(f"{step_num}\n{act}\n\n{obs}") memory_contexts.append("\n\n".join(blocks)) ``` -------------------------------- ### BashCodingMultiProcessEnv.__init__ Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Initializes the multi-process Bash coding environment with a seed, number of environments, group size, and per-worker resources. Creates Ray actors for each worker process. ```APIDOC ## `BashCodingMultiProcessEnv.__init__` ### Description Initializes the multi-process Bash coding environment. Creates `env_num * group_n` worker processes, each a Ray actor `BashCodingWorker`. When `is_train=False`, asserts `group_n == 1`. ### Method `__init__` ### Parameters - **seed** (int) - Required - Seed for worker initialization. - **env_num** (int) - Required - Number of distinct environments. - **group_n** (int) - Required - Number of workers per environment group. - **resources_per_worker** (dict) - Required - Resource specification for each worker (e.g., `{"num_cpus": 0.1}`). - **is_train** (bool) - Optional - Whether in training mode. Defaults to `True`. If `False`, `group_n` must be 1. - **env_kwargs** - Optional - Additional keyword arguments passed to each worker. ### Notes - `num_processes = env_num * group_n`. - Each worker is seeded with `seed + i // group_n`. - Raises `AssertionError` if `is_train=False` and `group_n != 1`. ``` -------------------------------- ### BashCodingWorker.__init__(seed, env_kwargs) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Initializes a BashCodingWorker instance with merged configuration from environment variables and provided kwargs. Sets up sandbox execution parameters and reward coefficients. ```APIDOC ## BashCodingWorker.__init__(seed, env_kwargs) ### Description Initializes a BashCodingWorker instance. The `env_kwargs` dictionary is merged with configuration from the `BASH_CODING_ENV_CONFIG_JSON` environment variable, with defaults winning over env var values. Sets up sandbox execution parameters and reward coefficients. ### Method Constructor ### Endpoint `BashCodingWorker(seed, env_kwargs)` ### Parameters - **seed** - Required - Seed for the environment - **env_kwargs** (`dict`) - Required - Configuration dictionary merged with environment variable config ### Configuration Keys (defaults) | Key | Default | Meaning | |-----|---------|---------| | `max_steps` | `10` | Episode step budget | | `timeout` | `10` | Per-command sandbox timeout (s) | | `execute_commands` | `False` | Gate for shell execution | | `max_output_chars` | `2000` | stdout+stderr truncation cap (kept tail) | | `max_diff_chars` | `1500` | file-diff truncation cap (kept head) | | `max_fs_diff_lines` | `50` | Diff line budget | | `max_created_preview_lines` | `20` | Preview lines for created files | | `unified_diff_context_lines` | `2` | `difflib.unified_diff` context | | `exec_backend` | `"sandbox"` | Execution backend name | | `exec_error_penalty` | `0.2` | Negative reward on errors (`-exec_error_penalty`) | | `answer_reward` | `1.0` | Terminal answer reward scale | | `progress_gain_coef` | `0.3` | Progress (fs-coverage) reward scale | | `use_model_evidence_gain` | `0` (bool(int)) | Gate semantic-similarity answer bonus | | `no_progress_on_answer` | `True` | Suppress progress gain on answer steps | | `require_answer_for_done` | `True` | Episode only ends via `submit_answer` | ``` -------------------------------- ### BashCodingMultiProcessEnv.step Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Steps the environment with the given actions. Asserts the number of actions matches the active number of processes. ```APIDOC ## `BashCodingMultiProcessEnv.step` ### Description Executes a step for each active process using the provided actions. Asserts that `len(actions) == active_n`. ### Method `step` ### Parameters - **actions** (list) - Required - List of actions to apply to each active process. ### Returns - **tuple** - `(obs_list, reward_list, done_list, info_list)`. ``` -------------------------------- ### Build and use multiprocess batch environment with build_bash_coding_envs Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Builds a BashCodingMultiProcessEnv with 4 environments and 2 groups per environment, using 0.1 CPU per worker. Resets with a list of task and reward_spec kwargs, then closes the environment. ```python from main_entry.cli_agent_bash_coding.env_package.bash_coding import build_bash_coding_envs envs = build_bash_coding_envs( seed=0, env_num=4, group_n=2, resources_per_worker={"num_cpus": 0.1}, is_train=True, max_steps=6, timeout=10, exec_backend="sandbox", bash_coding_harness="vanilla", ) obs_list, info_list = envs.reset(kwargs=[{"task": t, "reward_spec": spec} for t, spec in zip(tasks, specs)]) ... envs.close() ``` -------------------------------- ### Algorithm registry and lookup functions (algo/registry.py) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-algo-gigpo-hgpo.md Defines the _ALGORITHMS dictionary with entries for 'A3_algo' and 'hgpo', plus functions to retrieve and list algorithms. The lookup is case-insensitive and returns None for empty or unknown names. ```python _ALGORITHMS = { "A3_algo": BashCodingAlgorithm( name="A3_algo", compute_advantage=compute_A3_algo_advantage, compute_policy_loss=compute_policy_loss_A3_algo, ), "hgpo": BashCodingAlgorithm( name="hgpo", ), } def get_bash_coding_algorithm(name: str) -> BashCodingAlgorithm | None: key = str(name).strip().lower() if key == "": return None for reg_key, algo in _ALGORITHMS.items(): if str(reg_key).lower() == key: return algo return None def list_bash_coding_algorithms() -> tuple[str, ...]: return tuple(_ALGORITHMS.keys()) ``` -------------------------------- ### Configure inference checkpoint pair Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/README.md YAML configuration for inference. Fill model.checkpoint_pair with the trained checkpoint and base model path. Once set, the launcher enters TEST MODE automatically. Use vanilla for standard harness and sigma_reveal_rd for sigma-Reveal. ```yaml model: checkpoint_pair: ":" pretrained_model_path_default: "Qwen/Qwen3-14B" ``` -------------------------------- ### BashIntentActionSpace.__init__ Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md Initializes a BashIntentActionSpace instance with an optional tree-sitter parser. If no parser is provided, a default parser for bash is created. ```APIDOC ## BashIntentActionSpace.__init__ ### Description Initializes a BashIntentActionSpace instance. Accepts an optional tree-sitter Parser; if none is given, a default parser for the bash language is created. ### Method Constructor ### Signature ```python __init__(self, parser: Parser | None = None) -> None ``` ### Parameters - **parser** (Parser | None) - Optional. A tree-sitter Parser instance. If None, a default parser for bash is used. ### Notes Requires `tree-sitter` and `tree_sitter_bash` packages. ``` -------------------------------- ### Build sigma_reveal_rd harness Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-harness.md Constructs the sigma_reveal_rd harness and verifies its base flags. The build_text_obs call renders the task prompt template around the provided observation. ```python from main_entry.cli_agent_bash_coding.harness import build_bash_coding_harness harness = build_bash_coding_harness({"bash_coding_harness": "sigma_reveal_rd"}) assert harness.name == "sigma_reveal_rd" assert harness.sigma_reject_enabled is False # base flags obs = harness.build_text_obs(env_manager, ["first obs"], init=True) # obs[0] contains the task prompt template rendered around "first obs" ``` -------------------------------- ### Intent-distance clustering with BashIntentActionSpace Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-tooling.md Creates a BashIntentActionSpace instance and computes a distance matrix for a list of shell commands. Also prints the intent signature for a sample command. Useful for clustering similar bash intents. ```python from main_entry.cli_agent_bash_coding.tooling.action_space_similarity import BashIntentActionSpace space = BashIntentActionSpace() shells = ["ls | wc -l", "ls *.txt | wc -l", "echo $HOME", "grep error log.txt"] d = space.distance_matrix(shells) print(space.intent_signature("ls | wc -l")) # ['K:pipeline', 'V:ls', 'K:list', 'V:wc', 'W:wc', 'W:l', ...] ``` -------------------------------- ### BashCodingTrajectoryCollector class definition and initialization Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-rollout.md Defines the class with history delimiters and initializes the harness, evidence gain settings, and validation dumps. The harness is built from config.env, and evidence gain coefficient is set if enabled. ```python class BashCodingTrajectoryCollector(TrajectoryCollector): HISTORY_PREFIX = "Here is the history records:\n" HISTORY_SUFFIX = "\n\nYou are now at step " def __init__(self, config, tokenizer, processor=None): super().__init__(config=config, tokenizer=tokenizer, processor=processor) self.harness = build_bash_coding_harness( OmegaConf.to_container(config.env, resolve=True) ) self.enable_evidence_gain = bool(config.env.use_model_evidence_gain) self._validation_dataset_dump: list[str] = [] self._validation_prompt_dump: list[str] = [] if self.enable_evidence_gain: self.evidence_gain_coef = float(config.env.progress_gain_coef) else: self.evidence_gain_coef = None ``` -------------------------------- ### make_envs(config) - build train and val bash coding environments Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md Builds training and validation environments for bash coding tasks. Raises ValueError if env_name lacks 'bash_coding' or if env.rollout.n is not an int. Train envs use group_n = n (or 1 if n <= 0) and is_train=True; val envs use group_n=1, is_train=False, and seed+1000. Both share the same projection_f and are wrapped in BashCodingEnvironmentManager. ```python def make_envs(config): if "bash_coding" not in config.env.env_name.lower(): raise ValueError(...) if not isinstance(config.env.rollout.n, int): raise ValueError(...) group_n = config.env.rollout.n if config.env.rollout.n > 0 else 1 ... train_envs = build_bash_coding_envs(seed=config.env.seed, env_num=config.data.train_batch_size, group_n=group_n, resources_per_worker=..., is_train=True, **env_kwargs) val_envs = build_bash_coding_envs(seed=config.env.seed + 1000, env_num=config.data.val_batch_size, group_n=1, resources_per_worker=..., is_train=False, **env_kwargs) projection_f = partial(bash_coding_projection, enable_commit=uses_commit_action_schema(env_kwargs["bash_coding_harness"])) envs = BashCodingEnvironmentManager(train_envs, projection_f, config) val_envs = BashCodingEnvironmentManager(val_envs, projection_f, config) return envs, val_envs ``` -------------------------------- ### BashCodingEnvironmentManager __init__ method Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-env-manager.md Initializes the environment manager with memory selection based on BASH_CODING_ENABLE, method manager, harness, and sigma-related fields. Reads environment variables for semantic similarity URL and batch size. ```python class BashCodingEnvironmentManager(EnvironmentManagerBase): def __init__(self, envs, projection_f, config): enabled = str(os.environ.get("BASH_CODING_ENABLE", "0")).strip() == "1" self.memory = ResearchSimpleMemory() if enabled else BaselineSimpleMemory() self.method_manager = BaselineMethodManager.from_config(config) self.harness = build_bash_coding_harness( OmegaConf.to_container(config.env, resolve=True) ) self.sample_kwargs_list: List[Dict[str, Any]] = [] self.semantic_similarity_url = os.environ.get( "BASH_CODING_SEMANTIC_SIMILARITY_URL", "http://127.0.0.1:30003" ) self.semantic_similarity_batch_size = int( os.environ.get("BASH_CODING_SEMANTIC_SIMILARITY_BATCH_SIZE", "32") ) self._sigma_space: BashIntentActionSpace | None = ( BashIntentActionSpace() if else None ) self._sigma_seen: List[Set[Tuple[str, ...]]] = [] self._sigma_counts: List[Counter] = [] self._sigma_concentration_tau: float = float( getattr(config.env, "sigma_concentration_tau", 0.8) ) super().__init__(envs, projection_f, config) ``` -------------------------------- ### Repository directory tree Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/README.md Shows the top-level directory structure and key files for the agent_system, main_entry/cli_agent_bash_coding, verl, main_entry/ehrcon_rebuild, environ, and assets directories. Use this to navigate the codebase and locate the framework-agnostic RL agent base, the A3 domain code, and the vendored verl integration points. ```text agent_system/ # framework-agnostic multi-turn RL agent base environments/ base.py EnvironmentManagerBase, to_numpy __init__.py re-exports EnvironmentManagerBase + make_envs memory/ base.py BaseMemory (ABC) memory.py SimpleMemory, SearchMemory multi_turn_rollout/ rollout_loop.py TrajectoryCollector utils.py to_list_of_dict, torch_to_numpy, numpy_to_torch, process_image, adjust_batch, filter_group_data reward_manager/ episode.py EpisodeRewardManager main_entry/cli_agent_bash_coding/ # A3 domain code (the novel part) algo/ base.py BashCodingAlgorithm (dataclass) registry.py get_bash_coding_algorithm, list_bash_coding_algorithms A3_algo.py compute_A3_algo_advantage, compute_policy_loss_A3_algo, compute_policy_loss_gspo_sequence stable_ops.py normalization/fusion/trust-region helpers hgpo.py compute_hgpo_advantage (HGPO baseline) core_gigpo.py compute_gigpo_outcome_advantage (GiGPO baseline) env_manager.py BashCodingEnvironmentManager, make_envs env_package/bash_coding/ BashCodingWorker, BashCodingMultiProcessEnv (ray), build_bash_coding_envs, bash_coding_projection, sandbox_exec (run_in_unshare_sandbox, ExecResult, SandboxError) harness/ base.py BashCodingHarnessBase enhanced.py EnhancedBashCodingHarness (FILE_CHANGES) vanilla.py VanillaBashCodingHarness sigma_reveal_rd.py SigmaRevealRDBashCodingHarness (sigma-Reveal) manager.py build_bash_coding_harness (registry) memory.py / action_schema.py / cib_action_encoding.py / reward_provenance.py reward/ segmented_episode_reward.py SegmentedEpisodeRewardManager rstar.py apply_rstar_resampling rollout/ evidence_collector.py BashCodingTrajectoryCollector prompts/ bash_coding.py BASH_CODING_TEMPLATE, BASH_CODING_TEMPLATE_NO_HIS baseline_methods/ BaselineMethodManager, CommitCreditRuntime, RetroAgentRuntime, lats_infer.lats_step (LATS inference baseline) tooling/ action_space_similarity/bash_intent_action_space.py BashIntentActionSpace semantic_similarity/semantic_similarity.py OpenAI-compatible LLM judger client data_preprocess/ prepare_bash_coding.py (CLI), multibench_schema.py (normalize_example) eval/ eval_results_jsonl.py (CLI + lib), merge_results.py, align_results_to_test_parquet_rerun_eval.py infer/ pure_infer.py (CLI: API-server inference against a local SGLang server) train/ config.yaml variants, launch_bash_coding.sh launchers, load_config_yaml.py tooling/ add_dataset_field_to_parquet.py verl/ # vendored verl framework + A3 integration points protocol.py DataProto trainer/main_ppo.py main() Hydra entrypoint, CoupledReward, run_ppo, TaskRunner trainer/ppo/ray_trainer.py RayPPOTrainer (rstar hook, kl_oscillation_stop, HGPO adv) trainer/ppo/core_algos.py compute_grpo_outcome_advantage, compute_policy_loss, ... workers/actor/dp_actor.py DataParallelPPOActor (A3 dispatch) workers/fsdp_workers.py FSDP actor (A3 advantage dispatch) utils/torch_functional.py tokenize_and_postprocess_data, masked_whiten, ... main_entry/ehrcon_rebuild/ EHRCon benchmark reconstruction scripts environ/ conda env + apt bootstrap scripts assets/ paper figures ``` -------------------------------- ### Generate Freeform Texts for Reflection Baselines Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-rollout.md Generates freeform texts for reflection baselines using the _generate_freeform_texts method. Requires an actor_rollout_wg and meta_info from a gen_batch. Uses max_prompt_tokens=12000, hard_truncate_tokens=16000, and data_source='retroagent_reflection'. ```python texts = collector._generate_freeform_texts( prompts=["Reflect on the last episode..."], actor_rollout_wg=actor_rollout_wg, meta_info=gen_batch.meta_info, max_prompt_tokens=12000, hard_truncate_tokens=16000, data_source="retroagent_reflection", ) ``` -------------------------------- ### BashCodingWorker.step(action) Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/bash-coding-worker-env.md Executes a single action in the environment. Handles empty commands, format violations, answer submissions, and code execution with full reward and error handling. ```APIDOC ## BashCodingWorker.step(action: str) -> Tuple[str, float, bool, Dict] ### Description Executes a single action in the environment. Handles empty commands, format violations, answer submissions, and code execution with full reward and error handling. ### Method Method call ### Endpoint `step(action)` ### Parameters - **action** (`str`) - Required - The action string to execute ### Behavior | Input | Behavior | |-------|----------| | empty action | `("Error: Empty command is not allowed.", reward, done, info)` — `done = step_count >= max_steps`; if final step and `require_answer_for_done`, terminal-submission provenance (0) is added and failed-terminal progress is zeroed | | `FORMAT_VIOLATION_PREFIX` | `("Error: Format violation.", ...)` same structure | | `ANSWER_PREFIX` | `done=True`; reward from `_calculate_reward`; `won = is_correct = _check_task_completion(answer_text)`; info has `direct_answer=True`; incorrect answers zero the terminal progress | | anything else | `_execute_action(action)` → sandbox run; success/progress rewards; `done = step_count >= max_steps` (when `require_answer_for_done`) with final-step bookkeeping; `SandboxError` → `SandboxError: ` observation, `-exec_error_penalty`, `done=False`; other `Exception` → `EnvError: : ` observation, `-exec_error_penalty`, `done=True` | ### Info Dict Fields Every `info` dict contains at least: `won`, `error`, `returncode`, `stdout`, `stderr`, `work_dir`, `changed_files`, `diff`, `step`, `env_warnings`, `reward_provenance`, `reward_total`, plus harness extras via `harness.build_step_info(self)` and (exec path) `obs_truncation`. ### Returns - **Tuple[str, float, bool, Dict]** - Observation, reward, done flag, and info dict ``` -------------------------------- ### Training data flow Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/README.md Describes the training data flow from the verl trainer entrypoint through the environment manager, trajectory collector, reward manager, and advantage computation. Use this to understand how prompts, actions, sandbox execution, and rewards are orchestrated in the A3 training loop. ```text verl.trainer.main_ppo → TaskRunner.run → make_envs(config) builds ray BashCodingMultiProcessEnv workers wrapped by BashCodingEnvironmentManager; BashCodingTrajectoryCollector.multi_turn_loop drives prompt→action→sandbox-execute→observation rounds; per-step rewards and reward_provenance are accumulated into episode_rewards; SegmentedEpisodeRewardManager distributes the episode score over response token spans; the actor worker calls compute_A3_algo_advantage (dispatched by _get_enabled_bash_coding_algorithm, hyperparameters injected from actor.algorithm_* config keys) then compute_policy_loss_A3_algo. ``` -------------------------------- ### EnvironmentManagerBase class definition Source: https://github.com/hoyant-su/agentic-rl-a3/blob/main/_autodocs/api-reference/agent-system-environments.md Base class for multi-turn environment wrappers. Constructor stores envs, projection function, and config. Includes reset, step, build_text_obs, close, success_evaluator, _process_batch, and save_image methods. ```python class EnvironmentManagerBase: def __init__(self, envs, projection_f, config): self.envs = envs self.projection_f = projection_f self.config = config def reset(self, kwargs) -> Dict[str, Any]: obs, infos = self.envs.reset() return {"text": None, "image": obs, "anchor": None}, infos def step(self, text_actions: List[str]): actions, valids = self.projection_f(text_actions) next_obs, rewards, dones, infos = self.envs.step(actions) next_observations = {"text": None, "image": next_obs, "anchor": None} for i, info in enumerate(infos): info["is_action_valid"] = to_numpy(valids[i]) rewards = to_numpy(rewards) dones = to_numpy(dones) return next_observations, rewards, dones, infos def build_text_obs(self) -> List[str]: pass def close(self) -> None: self.envs.close() def success_evaluator(self, *args, **kwargs) -> Dict[str, np.ndarray]: total_infos = kwargs["total_infos"] total_batch_list = kwargs["total_batch_list"] batch_size = len(total_batch_list) success = defaultdict(list) for bs in range(batch_size): self._process_batch(bs, total_batch_list, total_infos, success) assert len(success["success_rate"]) == batch_size return {key: np.array(value) for key, value in success.items()} def _process_batch(self, batch_idx, total_batch_list, total_infos, success): for i in reversed(range(len(total_batch_list[batch_idx]))): batch_item = total_batch_list[batch_idx][i] if batch_item["active_masks"]: info = total_infos[batch_idx][i] won_value = float(info["won"]) success["success_rate"].append(won_value) return def save_image(self, image, step): path = os.path.join( os.path.dirname(__file__), os.path.join("images", self.config.env.env_name) ) if not os.path.exists(path): os.makedirs(path) path = os.path.join(path, f"step{step}.png") if isinstance(image, torch.Tensor): image = image.detach().cpu().numpy() if isinstance(image, np.ndarray): pass else: raise ValueError(f"Unsupported type: {type(image)})") if len(image.shape) == 4: image = image[0] if image.shape[0] == 3: image = np.transpose(image, (1, 2, 0)) if image.max() <= 1.0: image = image * 255 image = image.astype(np.uint8) from PIL import Image image = Image.fromarray(image) image.save(path) ```