### Setup Nightly Development Environment

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Set up a nightly development environment using the Makefile, installing Python 3.12 and the latest versions of torchrl and tensordict from git HEAD.

```bash
make setup-nightly
```

--------------------------------

### Setup Stable Development Environment

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Set up a stable development environment using the Makefile, installing Python 3.11 and pinned versions of torchrl and tensordict.

```bash
make setup-stable
```

--------------------------------

### Install Optional Dependency for Scaffold Decoration

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Install the 'promptsmiles' library to enable scaffold decoration and fragment linking functionalities.

```bash
pip3 install promptsmiles
```

--------------------------------

### Develop Install TorchRL

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Installs the torchrl library in development mode. This command should be run from the torchrl project directory.

```bash
cd /path/to/torchrl
python setup.py develop
```

--------------------------------

### Develop Install Tensordict

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Installs the tensordict library in development mode. This command should be run from the tensordict project directory.

```bash
cd /path/to/tensordict
python setup.py develop
```

--------------------------------

### Display Script Usage

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Use this command to view the help message and available options for the example scripts.

```bash
./run-example-scripts.sh --help
```

--------------------------------

### Develop Install Acegen Open

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Installs the acegen-open library in development mode. This command should be run from the acegen-open project directory.

```bash
cd /path/to/acegen-open
python setup.py develop
```

--------------------------------

### Run training scripts for fragment linking

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Executes training scripts using the linking configuration; requires promptsmiles installation.

```bash
python scripts/reinforce/reinforce.py --config-name config_linking
python scripts/a2c/a2c.py --config-name config_linking
python scripts/ppo/ppo.py --config-name config_linking
python scripts/reinvent/reinvent.py --config-name config_linking
python scripts/ahc/ahc.py --config-name config_linking
python scripts/dpo/dpo.py --config-name config_linking
python scripts/hill_climb/hill_climb.py --config-name config_linking
```

--------------------------------

### Develop Install MolScore

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Checks out the develop branch and installs the MolScore library in development mode. This command should be run from the MolScore project directory.

```bash
cd /path/to/MolScore
git checkout develop
python setup.py develop
```

--------------------------------

### Run training scripts for scaffold decoration

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Executes training scripts using the scaffold configuration; requires promptsmiles installation.

```bash
python scripts/reinforce/reinforce.py --config-name config_scaffold
python scripts/a2c/a2c.py --config-name config_scaffold
python scripts/ppo/ppo.py --config-name config_scaffold
python scripts/reinvent/reinvent.py --config-name config_scaffold
python scripts/ahc/ahc.py --config-name config_scaffold
python scripts/dpo/dpo.py --config-name config_scaffold
python scripts/hill_climb/hill_climb.py --config-name config_scaffold
```

--------------------------------

### Install Dependencies and Acegen Package

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Install project dependencies from 'requirements.txt' and the Acegen package itself. Use '-e' for editable mode during development.

```bash
pip install -r requirements.txt
pip install .
```

--------------------------------

### Scaffold Decoration Example

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

Illustrates how PromptSMILES handles scaffold decoration by elaborating a benzene ring at two meta-oriented substitution points.

```text
c1(\*)
cc(\*)
ccc1
```

```text
c1ccccc1Cl
```

```text
c1ccc(Cl)cc1Br
```

--------------------------------

### Execute scripts with custom configuration

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Runs an installed executable by specifying the path and name of the configuration file.

```bash
ppo.py --config-path=<path_to_config_dir> --config-name=<config_name.yaml>
```

--------------------------------

### Install Optional Dependencies for MolScore

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Install RDKit and MolScore if you plan to define custom scoring functions. Ensure RDKit version is compatible.

```bash
pip3 install rdkit>=2023.3.3
pip3 install MolScore
```

--------------------------------

### Superstructure Generation Example

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

Demonstrates superstructure generation where all atoms with available valence are specified as attachment points. Some sampled tokens may be the stop token.

```text
c1(\*)
c(\*)
c(\*)
c(\*)
c(\*)
c1(\*)
```

```text
c1ccccc1Cl
```

```text
c1cc(Cl)ccc1<EOS>
```

```text
c1c(Cl)cccc1F
```

```text
c1(Cl)cc(F)ccc1<EOS>
```

```text
c1(Cl)cc(F)ccc1
```

--------------------------------

### Configure Custom Model Factory

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Example configuration snippet for using an external model factory.

```yaml
...
model: example_model
custom_model_factory: my_module.custom_model_factory
...
```

--------------------------------

### Install MolScore Dependencies

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Installs the necessary dependencies for the MolScore library, including RDKit, dask, and various scientific computing packages.

```bash
pip3 install rdkit func_timeout dask distributed pystow zenodo_client matplotlib scipy pandas joblib seaborn molbloom Levenshtein
```

--------------------------------

### Default De Novo Generation Configuration

Source: https://context7.com/acellera/acegen-open/llms.txt

Example configuration file for de novo molecule generation, covering logging, environment, scoring, model, and optimizer settings.

```yaml
# config_denovo.yaml

# Logging configuration
experiment_name: acegen         # WandB/TensorBoard project name
agent_name: reinvent            # Agent identifier for logging
log_dir: results                # Directory to save results
logger_backend: wandb           # Options: wandb, tensorboard, null
seed: 101                       # Random seed (or list: [101, 102, 103])

# Environment configuration
num_envs: 128                   # Parallel SMILES generation
total_smiles: 10000             # Total molecules to generate

# Scoring function (MolScore integration)
molscore_mode: single           # Options: single, benchmark, curriculum
molscore_task: MolOpt:Albuterol_similarity  # Task specification
custom_task: null               # Custom scoring function (requires molscore_mode: null)

# PromptSMILES for constrained generation
prompt: null                    # Fixed SMILES prefix (e.g., "c1ccccc")
promptsmiles: null              # Scaffold/fragments (e.g., "c1c(*)cc(*)cc1")
promptsmiles_optimize: true     # Optimize prompt arrangement
promptsmiles_shuffle: true      # Randomize attachment points
promptsmiles_multi: false       # Multiple RL updates per completion

# Model architecture
model: gru                      # Options: gru, lstm, gpt2, mamba, llama2
custom_model_factory: null      # Path to custom model factory

# Optimizer configuration
lr: 0.0001
olds: 1.0e-08
weight_decay: 0.0

# Algorithm-specific (REINVENT)
sigma: 120                      # Score scaling factor

# Experience replay
experience_replay: true
replay_buffer_size: 100
replay_batch_size: 10
```

--------------------------------

### Clone Acegen-Open Repository

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Clone the Acegen-Open repository to your local machine. Navigate into the cloned directory to proceed with installation.

```bash
git clone https://github.com/Acellera/acegen-open.git
cd acegen-open
```

--------------------------------

### Install Base Dependencies

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Installs PyTorch nightly build for CUDA 12.1, along with tqdm, wandb, and hydra-core. Ensure your CUDA version is compatible.

```bash
pip3 install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121
```

```bash
pip3 install tqdm wandb hydra-core
```

--------------------------------

### Start Wandb Sweep

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/hyperparameter_optimisation_with_wandb.md

Command to initiate a Wandb sweep using a defined YAML configuration file. This command starts the sweep process and returns a sweep ID required for running agents.

```bash
wandb sweep ppo_sweep.yaml 

```

--------------------------------

### Fragment Linking Example

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

Shows fragment linking where different chemical substructures with one attachment point each are supplied, separated by a '.'. One fragment point is selected and completed, then the other is concatenated.

```text
C1CC1(*).c1(*)ccncc1
```

```text
C1CC1
```

```text
CCOCC
```

```text
C1CC1CCOCC + c1ccncc1
```

```text
C1CC1CCOCCc1ccncc1
```

--------------------------------

### Register GPT2 Model

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Example of adding a GPT2 model to the internal models dictionary.

```python
models = {
    "gpt2": (
        create_gpt2_actor,
        None,
        None,
        Path(__file__).resolve().parent.parent.parent / "priors" / "enamine_real_vocabulary.txt",
        Path(__file__).resolve().parent.parent.parent / "priors" / "gpt2_enamine_real.ckpt",
        None,
    )
}
```

--------------------------------

### Initialize SMILES Tokenizers

Source: https://context7.com/acellera/acegen-open/llms.txt

Initializes various SMILES tokenizers provided by ACEGEN for converting SMILES strings to tokens and vice-versa. Ensure necessary libraries like 'deepsmiles' and 'selfies' are installed if using their respective tokenizers.

```python
from acegen.vocabulary.tokenizers import (
    SMILESTokenizerChEMBL,
    SMILESTokenizerEnamine,
    SMILESTokenizerGuacaMol,
    AsciiSMILESTokenizer,
    DeepSMILESTokenizer,
    SELFIESTokenizer
)

# Standard ChEMBL tokenizer (handles brackets, ring numbers, Br/Cl)
tokenizer_chembl = SMILESTokenizerChEMBL()
tokens = tokenizer_chembl.tokenize("c1ccc(Cl)cc1")
print(tokens)  # ['c', '1', 'c', 'c', 'c', '(', 'Cl', ')', 'c', 'c', '1']
smiles = tokenizer_chembl.untokenize(tokens)
print(smiles)  # "c1ccc(Cl)cc1"

# Enamine tokenizer (atom-aware tokenization)
tokenizer_enamine = SMILESTokenizerEnamine()
tokens = tokenizer_enamine.tokenize("CC(=O)Oc1ccccc1")
print(tokens)  # ['C', 'C', '(', '=', 'O', ')', 'O', 'c', '1', 'c', 'c', 'c', 'c', 'c', '1']

# ASCII tokenizer (character-level, used by Llama2)
tokenizer_ascii = AsciiSMILESTokenizer(start_token="^", end_token="$")
tokens = tokenizer_ascii.tokenize("CCO", with_begin_and_end=True)
print(tokens)  # ['^', 'C', 'C', 'O', '$']

# DeepSMILES tokenizer (requires deepsmiles library)
tokenizer_deep = DeepSMILESTokenizer()
tokens = tokenizer_deep.tokenize("c1ccccc1")  # Converts to DeepSMILES first
smiles = tokenizer_deep.untokenize(tokens, convert_to_smiles=True)

# SELFIES tokenizer (requires selfies library)
tokenizer_selfies = SELFIESTokenizer()
tokens = tokenizer_selfies.tokenize("CCO")  # Converts to SELFIES tokens
smiles = tokenizer_selfies.untokenize(tokens, convert_to_smiles=True)
```

--------------------------------

### Run Wandb Agent

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/hyperparameter_optimisation_with_wandb.md

Command to start a Wandb agent that will run the specified program with hyperparameters sampled from the sweep. Replace <number_of_runs> with the desired number of agent runs and sweep_id with the ID obtained from the sweep command.

```bash
wandb agent --count <number_of_runs> sweep_id

```

--------------------------------

### Execute Training Script

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_scoring_function.md

Run the training process using the specified configuration file.

```bash
python acegen/scripts/reinvent/reinvent.py --config-name config_denovo
```

--------------------------------

### Initialize TokenEnv for Molecule Generation

Source: https://context7.com/acellera/acegen-open/llms.txt

Sets up a token-based reinforcement learning environment for step-by-step molecule generation. Requires wrapping with TransformedEnv for training compatibility.

```python
from acegen import TokenEnv
from torchrl.envs import TransformedEnv, InitTracker

# Create the token-based RL environment
env = TokenEnv(
    start_token=0,              # Index of start token in vocabulary
    end_token=1,                # Index of end token in vocabulary
    length_vocabulary=50,       # Total vocabulary size
    max_length=200,             # Maximum sequence length
    batch_size=128,             # Number of parallel SMILES generations
    device="cuda:0"             # Device for tensor operations
)

# Wrap with transforms for RL training
env = TransformedEnv(env)
env.append_transform(InitTracker())

# Reset environment to get initial observation
initial_obs = env.reset()
print(initial_obs.keys())  # ['observation', 'done', 'truncated', 'terminated', 'temperature', 'sequence', 'sequence_mask', 'action_mask']

# Step through environment with an action
tensordict = initial_obs.clone()
tensordict.set("action", torch.randint(0, 50, (128,)))
next_tensordict = env.step(tensordict)
```

--------------------------------

### Run REINFORCE Training Script

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the REINFORCE (vanilla policy gradient) training script with a default configuration.

```bash
python scripts/reinforce/reinforce.py --config-name config_denovo
```

--------------------------------

### Run PPO with MolScore Benchmark

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the PPO script with MolScore benchmark mode enabled.

```bash
python scripts/ppo/ppo.py --config-name config_denovo \
    molscore_mode=benchmark \
    molscore_task=MolOpt
```

--------------------------------

### Run PPO Training Scripts

Source: https://context7.com/acellera/acegen-open/llms.txt

Executes PPO (Proximal Policy Optimization) training scripts for de novo generation. Supports PPO+D variant with experience replay and shared actor-critic networks. Configuration is managed via YAML files and command-line overrides.

```bash
# De novo generation with PPO
python scripts/ppo/ppo.py --config-name config_denovo

# PPO with experience replay (PPO+D)
python scripts/ppo/ppo.py --config-name config_denovo \
    experience_replay=True \
    replay_buffer_size=100 \
    replay_batch_size=24

# PPO with shared actor-critic networks
python scripts/ppo/ppo.py --config-name config_denovo \
    shared_nets=True
```

--------------------------------

### Run DPO Training Script

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the Direct Preference Optimization (DPO) training script with a default configuration, overriding 'beta'.

```bash
python scripts/dpo/dpo.py --config-name config_denovo \
    beta=0.1
```

--------------------------------

### Run Fragment Linking

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the REINVENT script to perform fragment linking using the 'config_linking' configuration.

```bash
python scripts/reinvent/reinvent.py --config-name config_linking
```

--------------------------------

### Initialize the Token Environment

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Creates a TokenEnv instance configured for parallel molecule generation.

```python
from acegen.rl_env import TokenEnv

env =  TokenEnv(
    start_token=vocab1.start_token_index,
    end_token=vocab1.end_token_index,
    length_vocabulary=len(vocab1),
    batch_size=4, # Number of molecules to generate in parallel
)
```

--------------------------------

### Inspect Environment Specifications

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Prints the action and observation specifications for the token environment.

```python
print(env.full_action_spec)
```

```python
print(env.full_observation_spec)
```

--------------------------------

### Sync Environment with uv

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Use uv to synchronize project dependencies based on 'pyproject.toml' and 'uv.lock' for reproducible environments. Prefix commands with 'uv run'.

```bash
uv sync
```

--------------------------------

### Run training scripts for de novo generation

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Executes training scripts using the denovo configuration.

```bash
python scripts/reinforce/reinforce.py --config-name config_denovo
python scripts/a2c/a2c.py --config-name config_denovo
python scripts/ppo/ppo.py --config-name config_denovo
python scripts/reinvent/reinvent.py --config-name config_denovo
python scripts/ahc/ahc.py --config-name config_denovo
python scripts/dpo/dpo.py --config-name config_denovo
python scripts/hill_climb/hill_climb.py --config-name config_denovo
```

--------------------------------

### Dynamic Scaffold Generation from Command Line

Source: https://context7.com/acellera/acegen-open/llms.txt

Run de novo generation with a dynamic scaffold specified via command-line arguments.

```bash
python scripts/reinvent/reinvent.py --config-name config_denovo \
    promptsmiles="c1c(*)cc(*)cc1" \
    promptsmiles_optimize=true
```

--------------------------------

### Configure environment parameters

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Defines parallel generation capacity and total library size.

```yaml
# Environment configuration
num_envs: 128 # Number of SMILES to generate in parallel
total_smiles: 10_000 # Total number of SMILES to generate
```

--------------------------------

### Configure data replay

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Settings for experience replay buffer size and batching.

```yaml
# Data replay configuration
experience_replay: True
replay_buffer_size: 100  # Size of the replay buffer in number of molecules
replay_batch_size: 10  # Size of the batch that will be sampled in every iteration to mix with the data generated by the RL agent.
```

--------------------------------

### Generate Environment Rollout

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Executes a full rollout of the environment for a specified number of steps using a given policy.

```python
rollout = env.rollout(max_steps=100, policy=policy)
print(rollout)
```

--------------------------------

### Scaffold Decoration Configuration

Source: https://context7.com/acellera/acegen-open/llms.txt

Configuration for scaffold decoration using PromptSMILES, specifying the scaffold and optimization parameters.

```yaml
# Scaffold decoration configuration (config_scaffold.yaml)
# Decorate a piperazine scaffold at two positions
promptsmiles: "N1(*)CCN(CC1)CCCCN(*)"
promptsmiles_optimize: true     # Find optimal SMILES arrangement
promptsmiles_shuffle: true      # Randomize attachment point selection
promptsmiles_multi: false       # Single update for complete molecule
```

--------------------------------

### Fragment Linking Configuration

Source: https://context7.com/acellera/acegen-open/llms.txt

Configuration for fragment linking using PromptSMILES, specifying the fragments and optimization parameters.

```yaml
# Fragment linking configuration (config_linking.yaml)
# Link cyclopropyl and pyridine fragments
promptsmiles: "C1CC1(*).c1(*)ccncc1"
promptsmiles_optimize: true
promptsmiles_shuffle: true
promptsmiles_multi: false
promptsmiles_scan: false        # Scan through linker positions
```

--------------------------------

### PromptSMILES Configuration for Fragment Linking

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

YAML configuration for PromptSMILES to link two fragments separated by '.', enabling optimization and shuffling.

```yaml
promptsmiles: "C1CC1(*).c1(*)ccncc1"
promptsmiles_optimize: True
promptsmiles_shuffle: True
promptsmiles_multi: False
```

--------------------------------

### PromptSMILES Configuration for Scaffold Decoration

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

YAML configuration for PromptSMILES to handle meta-substituted benzene, enabling optimization and shuffling of generated arrangements.

```yaml
promptsmiles: "c1c(*)cc(*)cc1"
promptsmiles_optimize: True
promptsmiles_shuffle: True
promptsmiles_multi: False
```

--------------------------------

### Create Actor and Critic Models

Source: https://context7.com/acellera/acegen-open/llms.txt

Uses factory functions to instantiate TorchRL-compatible models. Returns separate training and inference models that share weights.

```python
from acegen.models import models, create_gru_actor, create_gru_critic, create_gru_actor_critic
from acegen.vocabulary import Vocabulary
import torch

# List available pre-trained models
print(list(models.keys()))  # ['gru', 'gru_chembl34', 'lstm', 'gpt2', 'llama2', 'mamba', ...]

# Get model components for a specific architecture
create_actor, create_critic, create_actor_critic, voc_path, ckpt_path, tokenizer = models["gru"]

# Load vocabulary and create models
vocabulary = Vocabulary.load(voc_path, tokenizer=tokenizer)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Create actor models (training and inference versions)
actor_training, actor_inference = create_actor(vocabulary_size=len(vocabulary))

# Load pre-trained weights
ckpt = torch.load(ckpt_path, map_location=device, weights_only=True)
actor_inference.load_state_dict(ckpt)
actor_inference = actor_inference.to(device)
actor_training = actor_training.to(device)

# Create critic models for actor-critic algorithms (PPO, A2C)
critic_training, critic_inference = create_critic(vocabulary_size=len(vocabulary))
```

--------------------------------

### Run REINVENT Training Scripts

Source: https://context7.com/acellera/acegen-open/llms.txt

Executes REINVENT training scripts for de novo generation, scaffold decoration, and fragment linking. Configuration can be managed via YAML files and overridden from the command line. Custom scoring functions and MolScore benchmark tasks are also supported.

```bash
# De novo generation with REINVENT
python scripts/reinvent/reinvent.py --config-name config_denovo

# Scaffold decoration with REINVENT
python scripts/reinvent/reinvent.py --config-name config_scaffold

# Fragment linking with REINVENT
python scripts/reinvent/reinvent.py --config-name config_linking

# Override configuration from command line
python scripts/reinvent/reinvent.py --config-name config_denovo \
    total_smiles=50000 \
    num_envs=256 \
    model=gpt2 \
    sigma=100 \
    lr=0.0005

# Run with custom scoring function
python scripts/reinvent/reinvent.py --config-name config_denovo \
    molscore_mode=null \
    custom_task=QED

# Run with MolScore benchmark task
python scripts/reinvent/reinvent.py --config-name config_denovo \
    molscore_mode=single \
    molscore_task=MolOpt:Zaleplon_MPO
```

--------------------------------

### Define PPO Sweep Configuration

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/hyperparameter_optimisation_with_wandb.md

YAML file to configure a Wandb sweep for the PPO algorithm. Specifies the program to run, the sweep method, and the hyperparameters to search over. Use this to define the search space for your hyperparameter optimization.

```yaml
program: ppo.py 
method: random 
parameters:
  num_envs:
    values: [32, 64, 128, 256]
  critic_coef:
    values: [0.1, 0.25, 0.5]
  entropy_coef:
    values: [0.01, 0.05]
  ppo_epochs:
    values: [1, 2, 3]

command:
  - ${env}
  - python
  - ${program}
  - ${args_no_hyphens} # Necessary for use with hydra as used in ACEGEN

```

--------------------------------

### Configure optimizer parameters

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Sets training hyperparameters like learning rate and weight decay.

```yaml
# Optimizer configuration
lr: 0.0001
eps: 1.0e-08
weight_decay: 0.0
```

--------------------------------

### Full MolScore Benchmark Configuration

Source: https://context7.com/acellera/acegen-open/llms.txt

Configure MolScore to run a full benchmark across all 22 MolOpt tasks.

```yaml
# Run full benchmark
molscore_mode: benchmark
molscore_task: MolOpt  # Runs all 22 MolOpt tasks
```

--------------------------------

### Configure logging settings

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Settings for experiment tracking, logging backends, and random seeds.

```yaml
# Logging configuration
experiment_name: acegen # Used for logging to WandB or TensorBoard
agent_name: reinvent # Used for logging to WandB or TensorBoard
log_dir: results # Path where results will be saved
logger_backend: null  # WandB, TensorBoard, or null
seed: 101 # Set the seed of each experiment. Multiple seeds can be provided as a list for sequential experiments, e.g., [101, 102, 103]
```

--------------------------------

### Configure Promptsmiles

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Sets the prompt for generation tasks.

```yaml
# Promptsmiles configuration
prompt: null  # e.g. c1ccccc
```

--------------------------------

### Create a Vocabulary from a List of Characters

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Initializes a vocabulary by mapping a provided list of characters to indices.

```python
from acegen.vocabulary import Vocabulary

chars = ["START", "END", "(", ")", "1", "=", "C", "N", "O"]
chars_dict = {char: index for index, char in enumerate(chars)}
vocab1 = Vocabulary.create_from_dict(chars_dict, start_token="START", end_token="END")
```

--------------------------------

### Generate Molecule Library with SAC

Source: https://github.com/acellera/acegen-open/blob/main/scripts/sac/README.md

Execute this command once both the actor and critic are pretrained to generate a library of molecules. The results may not be comparable to on-policy methods.

```python
python sac.py
```

--------------------------------

### Interact with the Token Environment

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Resets the environment to generate an initial observation.

```python
initial_td = env.reset() 
print(initial_td)
```

--------------------------------

### Initialize Random Policy and Apply to TensorDict

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Uses a RandomPolicy from TorchRL to select an action based on the environment's action specification and updates the TensorDict.

```python
from torchrl.collectors import RandomPolicy

policy = RandomPolicy(env.full_action_spec)
initial_td = policy(initial_td)
print(initial_td)
```

--------------------------------

### Create Combined Actor-Critic Model

Source: https://context7.com/acellera/acegen-open/llms.txt

Initializes a combined actor-critic model with specified architectural parameters.

```python
actor_train, actor_inf, critic_train, critic_inf = create_actor_critic(
    vocabulary_size=len(vocabulary),
    embedding_size=256,
    hidden_size=512,
    num_layers=3,
    dropout=0.0
)
```

--------------------------------

### Run A2C Training Script

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the Advantage Actor-Critic (A2C) training script with a default configuration.

```bash
python scripts/a2c/a2c.py --config-name config_denovo
```

--------------------------------

### Execute Environment Step

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Advances the environment state by one step using the current TensorDict and retrieves the next state.

```python
initial_td = env.step(initial_td)
print(initial_td)
next_td = initial_td.get("next")
```

--------------------------------

### Run AHC with LibINVENT Benchmark Task

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the AHC script using the 'config_scaffold' configuration for a specific LibINVENT benchmark task.

```bash
python scripts/ahc/ahc.py --config-name config_scaffold \
    molscore_task=LibINVENT_Exp1:DRD2_SelRF_SubFilt_DF
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Create a new Conda environment named 'acegen' with Python 3.10 and activate it. This isolates project dependencies.

```bash
conda create -n acegen python=3.10 -y
conda activate acegen
```

--------------------------------

### Custom Model Registration

Source: https://context7.com/acellera/acegen-open/llms.txt

Demonstrates how to register custom neural network models by providing model factories. These factories should return TorchRL-compatible actor and critic modules.

```python
from acegen.models import register_model, models
from tensordict.nn import TensorDictModule, TensorDictSequential
from torchrl.modules import ProbabilisticActor
from torchrl.envs import ExplorationType
import torch
import torch.nn as nn
from pathlib import Path
```

--------------------------------

### Run Hill Climbing Baseline Script

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the non-RL Hill Climbing baseline script with a default configuration.

```bash
python scripts/hill_climb/hill_climb.py --config-name config_denovo
```

--------------------------------

### Define general configuration file structure

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

The high-level layout of an ACEGEN YAML configuration file.

```yaml
# Logging configuration
...

# Environment configuration
...

# Scoring function
...

# Promptsmiles configuration
...

# Model architecture
...

# Optimizer configuration
...

# Algorithm configuration
...

# Data replay configuration
...
```

--------------------------------

### Pretrain SAC Critic

Source: https://github.com/acellera/acegen-open/blob/main/scripts/sac/README.md

Use this command to pretrain the critic after the actor/policy has been pretrained. This script is provided for reference and has not been sufficiently tested.

```python
python pretrain_sac.py
```

--------------------------------

### Override PPO Hyperparameters

Source: https://context7.com/acellera/acegen-open/llms.txt

Override PPO hyperparameters directly from the command line for custom training runs.

```bash
python scripts/ppo/ppo.py --config-name config_denovo \
    gamma=0.99 \
    lmbda=0.95 \
    ppo_epochs=4 \
    ppo_clip=0.2 \
    entropy_coef=0.01 \
    critic_coef=0.5
```

--------------------------------

### Run Scaffold Decoration

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the AHC script to perform scaffold decoration using the 'config_scaffold' configuration.

```bash
python scripts/ahc/ahc.py --config-name config_scaffold
```

--------------------------------

### Create GPT2 Critic

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Implementation of a critic model using TensorDictModule for compatibility with PPO and A2C training scripts.

```python
def create_gpt2_critic(
    vocabulary_size: int,
):
    """Create a GPT2 critic for language modeling."""
    # Define transformer
    config = GPT2Config()  # Original GPT2 configuration, can be customized
    config.vocab_size = vocabulary_size
    lm = GPT2(config)

    # Wrap the transformer in a TensorDictModule to make TensorDict compatible
    lm_training = TensorDictModule(
        lm.set_train_mode(True),
        in_keys=["sequence", "sequence_mask"],
        out_keys=["features"],
    )
    lm_inference = TensorDictModule(
        lm,
        in_keys=["sequence", "sequence_mask"],
        out_keys=["features"],
    )

    # Define final layer and also make it a TensorDictModule
    lm_head = TensorDictModule(
        nn.Linear(
            config.n_embd,
            1,
            bias=False,
        ),
        in_keys=["features"],
        out_keys=["state_value"],
    )

    # Concatenate lm and head, similar to torch.nn.Sequential
    # Critic does not need to be probabilistic, so we can return directly
    critic_training = TensorDictSequential(lm_training, lm_head)
    critic_inference = TensorDictSequential(lm_inference, lm_head)
    return critic_training, critic_inference
```

--------------------------------

### Fragment Linking with NLL Evaluation

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/using_promptsmiles.md

Demonstrates fragment linking where the second fragment is inserted at every point in the generated linker, and the likelihood of the model generating this de novo is evaluated using negative log-likelihood (NLL).

```text
C1CC1
```

```text
CCOCC
```

```text
C1CC1C(c1ccncc1)COCC  NLL=24.3
```

```text
C1CC1CC(c1ccncc1)OCC  NLL=25.6
```

```text
C1CC1CCO(c1ccncc1)CC  NLL=60.8
```

```text
C1CC1CCOC(c1ccncc1)C  NLL=23.2
```

```text
C1CC1CCOCC(c1ccncc1)  NLL=20.2 (selected insertion point)
```

--------------------------------

### Curriculum Learning MolScore Configuration

Source: https://context7.com/acellera/acegen-open/llms.txt

Configure MolScore for curriculum learning, specifying the sequential task curriculum.

```yaml
# Curriculum learning (sequential tasks)
molscore_mode: curriculum
molscore_task: GuacaMol:curriculum_1
```

--------------------------------

### Create a Tensordict-compatible module

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Wraps a standard PyTorch linear layer with TensorDictModule to process data stored in a TensorDict.

```python
import torch
from tensordict import TensorDict
from tensordict.nn import TensorDictModule

data = TensorDict({
    "key_1": torch.ones(3, 4),
    "key_2": torch.zeros(3, 4, dtype=torch.bool),
}, batch_size=[3])

# Define a simple module
module = torch.nn.Linear(4, 5)

# Make it Tensordict-compatible
td_module = TensorDictModule(module, in_keys=["key_1"], out_keys=["key_3"])

# Apply the module to the Tensordict data
data = td_module(data)
print(data)

# Output
TensorDict(
    fields={
        key_1: Tensor(shape=torch.Size([3, 4]), device=cpu, dtype=torch.float32, is_shared=False),
        key_2: Tensor(shape=torch.Size([3, 4]), device=cpu, dtype=torch.bool, is_shared=False),
        key_3: Tensor(shape=torch.Size([3, 5]), device=cpu, dtype=torch.float32, is_shared=False)},
    batch_size=torch.Size([3]),
    device=None,
    is_shared=False,
)
```

--------------------------------

### Configure model architecture

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Selects the generative model type or provides a custom model factory path.

```yaml
# Model architecture
model: gru # gru, lstm, or gpt2, mamba, llama2, etc
# The default prior varies for each model. Refer to the README file in the root directory for more information.
# The default vocabulary varies for each prior. Refer to the README file in the root directory for more information.
custom_model_factory: null # Path to a custom model factory (e.g. my_module.create_model)
```

--------------------------------

### Override configuration parameters via command line

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Passes additional YAML parameters directly through the command line.

```bash
ppo.py --config-path=<path_to_config_dir> --config-name=<config_name.yaml> total_smiles=100
```

--------------------------------

### Wrap model in ProbabilisticActor

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Configures the GPT-2 model as a TensorDict-compatible module and wraps it in a ProbabilisticActor for action sampling.

```python
from tensordict.nn import TensorDictModule, TensorDictSequential
from torchrl.envs import ExplorationType
from torchrl.modules import ProbabilisticActor


def create_gpt2_actor(
    vocabulary_size: int,
):
    # Define transformer
    config = GPT2Config()  # Original GPT2 configuration, can be customized
    config.vocab_size = vocabulary_size
    lm = GPT2(config)    

    # Wrap the transformer in a TensorDictModule to make TensorDict compatible
    lm_training = TensorDictModule(
        lm.set_train_mode(True),
        in_keys=["sequence", "sequence_mask"],
        out_keys=["features"],
    )
    lm_inference = TensorDictModule(
        lm,
        in_keys=["sequence", "sequence_mask"],
        out_keys=["features"],
    )

    # Define final layer and also make
    lm_head = TensorDictModule(
        nn.Linear(config.n_embd, vocabulary_size, bias=False),
        in_keys=["features"],
        out_keys=["logits"],
    )

    # Concatenate lm and head, similar to torch.nn.Sequential
    policy_training = TensorDictSequential(lm_training, lm_head)
    policy_inference = TensorDictSequential(lm_inference, lm_head)

    # To make the actor probabilistic, wrap the policy in a ProbabilisticActor
    # This module will take care of sampling and computing log probabilities
    probabilistic_policy_training = ProbabilisticActor(
        module=policy_training,
        in_keys=["logits"],
        out_keys=["action"],
        distribution_class=torch.distributions.Categorical,
        return_log_prob=return_log_prob,
        default_interaction_type=ExplorationType.RANDOM,
    )
    probabilistic_policy_inference = ProbabilisticActor(
        module=policy_inference,
        in_keys=["logits"],
        out_keys=["action"],
        distribution_class=torch.distributions.Categorical,
        return_log_prob=True,
        default_interaction_type=ExplorationType.RANDOM,
    )
    return probabilistic_policy_training, probabilistic_policy_inference
```

--------------------------------

### Register GPT2 with Critic

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Updated models dictionary registration including the critic model and a tokenizer.

```python
models = {
    "gpt2": (
        create_gpt2_actor,
        create_gpt2_critic,
        None,
        Path(__file__).resolve().parent.parent.parent / "priors" / "enamine_real_vocabulary.txt",
        Path(__file__).resolve().parent.parent.parent / "priors" / "gpt2_enamine_real.ckpt",
        SMILEStokenizer2(), # Constratined generation tasks require a tokenizer
    )
}
```

--------------------------------

### Run AHC Training Script with Hyperparameters

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the Augmented Hill-Climb (AHC) training script, overriding 'topk' and 'sigma' hyperparameters.

```bash
python scripts/ahc/ahc.py --config-name config_denovo \
    topk=0.5 \
    sigma=60
```

--------------------------------

### Print AceGen Reward Spec

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Prints the full reward specification of the AceGen environment, showing the unbounded continuous tensor spec for rewards.

```python
print(env.full_reward_spec)
```

--------------------------------

### Print AceGen Done Spec

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Prints the full done specification of the AceGen environment, detailing discrete tensor specs for done, truncated, and terminated states.

```python
print(env.full_done_spec)
```

--------------------------------

### Define Model Mapping Structure

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

The expected structure for the models dictionary in /acegen/__init__.py.

```python
models = {
    "example_model": (
        create_actor_method: Callable # A method to create the actor model
        create_critic_method: Callable # A method to create the critic model (Optional)
        create_actor_critic_method: Callable # A method to create the actor-critic model (Optional)
        vocabulary_file_path: Path # The path to the vocabulary file
        weights_file_path: Path # The path to the weights file (Optional)
        tokenizer: Tokenizer # The tokenizer to use for the model (Optional)
    )
}
```

--------------------------------

### Run Nightly Tests

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Execute the test suite using pytest within the 'acegen-nightly' Conda environment.

```bash
make test-nightly
```

--------------------------------

### Custom MolScore Configuration File

Source: https://context7.com/acellera/acegen-open/llms.txt

Configure MolScore to use a custom scoring configuration file specified by its path.

```yaml
# Custom MolScore configuration file
molscore_mode: single
molscore_task: /path/to/custom_config.json
```

--------------------------------

### Create Conda Environment

Source: https://github.com/acellera/acegen-open/blob/main/tests/check_scripts/README.md

Creates a new Conda environment named 'acegen-scripts-check' with Python 3.10. This is the first step in setting up the project dependencies.

```bash
conda create -n acegen-scripts-check python=3.10 -y
```

--------------------------------

### Load a Vocabulary from a State Dictionary

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Restores a previously saved vocabulary using its state dictionary.

```python
state_dict = vocab2.state_dict()
vocab3 = Vocabulary.load_state_dict(state_dict)
```

--------------------------------

### Run REINVENT with Specific MolScore Task

Source: https://context7.com/acellera/acegen-open/llms.txt

Execute the REINVENT script for single objective optimization on the DRD2 task.

```bash
python scripts/reinvent/reinvent.py --config-name config_denovo \
    molscore_mode=single \
    molscore_task=MolOpt:DRD2
```

--------------------------------

### Define Custom GRU Actor Model

Source: https://context7.com/acellera/acegen-open/llms.txt

Defines a custom GRU-based actor model and registers it with ACEGEN. Requires PyTorch and TensorDict.

```python
from pathlib import Path

import torch
from torch import nn

from tensordict.nn import TensorDictModule
from torchrl.modules import ProbabilisticActor, ExplorationType

from acegen.models.registry import register_model


def create_custom_gru_actor(vocabulary_size: int):
    """Create a custom GRU-based actor."""

    class CustomGRU(nn.Module):
        def __init__(self, vocab_size, hidden_size=256):
            super().__init__()
            self.embedding = nn.Embedding(vocab_size, hidden_size)
            self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
            self.head = nn.Linear(hidden_size, vocab_size)

        def forward(self, x):
            embedded = self.embedding(x)
            output, _ = self.gru(embedded)
            return self.head(output[:, -1, :])

    model = CustomGRU(vocabulary_size)

    # Wrap for TensorDict compatibility
    td_module = TensorDictModule(
        model,
        in_keys=["observation"],
        out_keys=["logits"]
    )

    # Create probabilistic actor
    actor = ProbabilisticActor(
        module=td_module,
        in_keys=["logits"],
        out_keys=["action"],
        distribution_class=torch.distributions.Categorical,
        return_log_prob=True,
        default_interaction_type=ExplorationType.RANDOM
    )

    return actor, actor  # Return training and inference versions

# Create factory function that returns full model specification
def custom_model_factory():
    return (
        create_custom_gru_actor,  # Actor factory
        None,                      # Critic factory (optional)
        None,                      # Actor-critic factory (optional)
        Path("path/to/vocabulary.txt"),
        Path("path/to/checkpoint.ckpt"),
        None                       # Tokenizer (optional)
    )

# Register the model
register_model("custom_gru", custom_model_factory)

# Now use in config: model: custom_gru
```

--------------------------------

### Print AceGen Observation Spec

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Prints the full observation specification of the AceGen environment, including discrete tensor specs for observation, sequence, and sequence mask.

```python
print(env.full_observation_spec)
```

--------------------------------

### Run Stable Tests

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Execute the test suite using pytest within the 'acegen-stable' Conda environment.

```bash
make test-stable
```

--------------------------------

### Configure scoring function

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Settings for MolScore integration or custom task definitions.

```yaml
# Scoring function
molscore_mode: single # single (run a single objective), benchmark (run multiple), or curriculum (run a sequence)
molscore_task: MolOpt:Albuterol_similarity # accepts task configuration path (JSON), benchmark (preset only), or curriculum task (preset only). In this case it select a specific ibjective from the MolOpt benchmark
custom_task: null # Requires molscore to be set to null. Allows to select a custom scoting function.
```

--------------------------------

### Create a Vocabulary from SMILES Strings

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Generates a vocabulary from a list of SMILES strings using a specified tokenizer.

```python
from acegen.vocabulary import SMILESTokenizerChEMBL

smiles_list = [
    "CCO",  # Ethanol (C2H5OH)
    "CCN(CC)CC",  # Triethylamine (C6H15N)
    "CC(=O)OC(C)C",  # Diethyl carbonate (C7H14O3)
    "CC(C)C",  # Isobutane (C4H10)
    "CC1=CC=CC=C1",  # Toluene (C7H8)
]

vocab2 = Vocabulary.create_from_strings(
    smiles_list, start_token="START", end_token="END", tokenizer=SMILESTokenizerChEMBL(),
)
```

--------------------------------

### Add Recurrent State to Observation with TensorDictPrimer

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/understanding_the_token_environment.md

Demonstrates extending the AceGen environment by adding a 'recurrent_state' field to the observation TensorDict using TorchRL's TensorDictPrimer transform. This is useful for recurrent models.

```python
from torchrl.envs.transforms import TensorDictPrimer
from torchrl.data.tensor_specs import UnboundedContinuousTensorSpec
from torchrl.envs import TransformedEnv

my_rnn_transform = TensorDictPrimer(
    {
        "recurrent_state": UnboundedContinuousTensorSpec(shape=(1, 10)),
    }
)

env = TransformedEnv(env, my_rnn_transform)
obs = env.reset()
print(obs)
```

--------------------------------

### Configure REINVENT algorithm

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/breaking_down_configuration_files.md

Defines hyperparameters specific to the REINVENT algorithm.

```yaml
# Reinvent configuration
sigma: 120
```

--------------------------------

### Single Objective MolScore Optimization

Source: https://context7.com/acellera/acegen-open/llms.txt

Configure MolScore for single objective optimization, specifying the task for Albuterol similarity.

```yaml
# Single objective optimization
molscore_mode: single
molscore_task: MolOpt:Albuterol_similarity
```

--------------------------------

### Clean Up Conda Environments

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Remove both the 'acegen-stable' and 'acegen-nightly' Conda environments created by the Makefile.

```bash
make clean-envs
```

--------------------------------

### Define a custom GPT-2 model

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Implements a torch.nn.Module that switches between training and inference modes to handle sequence masking.

```python
import torch
from torch import nn
from transformers import GPT2Config, GPT2Model

class GPT2(nn.Module):

    def __init__(self, config=None):
        super(GPT2, self).__init__()
        self.feature_extractor = GPT2Model(config) if config is not None else None        
        self._train_mode = False
        
    @property
    def train_mode(self):
        return self._train_mode
    
    def set_train_mode(self, train_mode: bool = True):
        if train_mode is self._train_mode:
            return self
        out = GPT2()
        out.feature_extractor = self.feature_extractor
        out._train_mode = train_mode
        return out

    def forward(self, sequence, sequence_mask):

        out = self.feature_extractor(
            input_ids=sequence,
            attention_mask=sequence_mask.long(),
        ).last_hidden_state

        if self.train_mode is False:  # If inference, return only last token set to True by the sequence_mask
            obs_length = sequence_mask.sum(-1)
            out = out[torch.arange(len(out)), obs_length.to(torch.int64) - 1]

        return out
```

--------------------------------

### Run Tests in Active Environment

Source: https://github.com/acellera/acegen-open/blob/main/README.md

Execute the test suite using pytest within the currently active Conda environment.

```bash
make test
```

--------------------------------

### Define External Model Factory

Source: https://github.com/acellera/acegen-open/blob/main/tutorials/adding_custom_model.md

Alternative method to register a model by defining a factory function that returns the required tuple.

```python
def example_model_fectory():
    return (
        create_gpt2_actor,
        None,
        None,
        Path(__file__).resolve().parent.parent.parent / "priors" / "enamine_real_vocabulary.txt",
        Path(__file__).resolve().parent.parent.parent / "priors" / "gpt2_enamine_real.ckpt",
        None,
    )
```