### Install Acme with Example Environments

Source: https://github.com/google-deepmind/acme/blob/master/README.md

Install Acme along with common reinforcement learning environments such as gym, dm_control, and bsuite.

```bash
pip install dm-acme[envs]
```

--------------------------------

### Install Acme with JAX, TensorFlow, and Environments

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Installs the dm-acme library with support for JAX, TensorFlow, and environments using pip.

```bash
%pip install git+https://github.com/deepmind/acme.git#egg=dm-acme[jax,tf,envs]
```

--------------------------------

### Install Acme from GitHub with Full Dependencies

Source: https://github.com/google-deepmind/acme/blob/master/README.md

Install the latest development version of Acme directly from the GitHub repository, including JAX, TensorFlow, testing utilities, and example environments.

```bash
pip install .[jax,tf,testing,envs]
```

--------------------------------

### Create Reverb server and client

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Initializes a Reverb server with replay tables and creates a client to connect to it. Use to manage the replay buffer in a distributed or single-process setup.

```python
replay_server = reverb.Server(replay_tables, port=None)
replay_client = reverb.Client(f'localhost:{replay_server.port}')
```

--------------------------------

### Install Acme with JAX and TensorFlow Support

Source: https://github.com/google-deepmind/acme/blob/master/README.md

Install the core dm-acme library along with JAX and TensorFlow dependencies. This is recommended for running most included agents.

```bash
pip install dm-acme[jax,tf]
```

--------------------------------

### Adder Usage Example

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Demonstrates the usage of an Adder for collecting agent experiences. It shows resetting the environment, adding the first timestep, and then iteratively acting, stepping the environment, and adding subsequent steps.

```python
# Reset the environment and add the first observation.
timestep = env.reset()
adder.add_first(timestep)

while not timestep.last():
  # Generate an action from the policy and step the environment.
  action = my_policy(timestep)
  timestep = env.step(action)

  # Add the action and the resulting timestep.
  adder.add(action, next_timestep=timestep)
```

--------------------------------

### Run DQN agent on Atari

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a DQN agent, a classic benchmark agent for Atari environments. This snippet shows the basic setup for running the agent.

```python
import acme
import acme.agents.tf.dqn as dqn

# Environment
env = gym.make('Pong-v0')

# Agent
agent = dqn.DQN(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run DQfD agent with demonstration data

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of the DQfD (Deep Q-learning from Demonstrations) agent. It runs on hard-exploration tasks using demonstration data, mixing offline and online learning.

```python
import acme
import acme.agents.tf.dqfd as dqfd

# Environment
env = bsuite.load_and_wrapper('deep-sea', average_discount=0.99)

# Agent
agent = dqfd.DQfD(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run SAC Baseline Example

Source: https://github.com/google-deepmind/acme/blob/master/examples/baselines/README.md

Use this command to run the SAC baseline in distributed mode on the Hopper environment. Specify the environment name and the total number of steps.

```bash
cd examples/baselines/rl_continuous
python run_sac.py --run_distributed=True --env_name=gym:Hopper-v2 --num_steps=100_000
```

--------------------------------

### Run BCQ agent offline

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example implementation of the BCQ (Batch-Constrained deep Q-learning) agent for offline learning. It learns from external data.

```python
import acme
import acme.agents.tf.bcq as bcq

# Environment
env = dm_control.make('cartman-push-v0')

# Agent
agent = bcq.BCQ(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run DQN agent on Behaviour Suite

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of an off-policy DQN agent running on tasks from the Behaviour Suite for Reinforcement Learning.

```python
import acme
import acme.agents.tf.dqn as dqn

# Environment
env = bsuite.load_and_wrapper('catch', average_discount=0.99)

# Agent
agent = dqn.DQN(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run Impala agent on Behaviour Suite

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of an on-policy Impala agent running on tasks from the Behaviour Suite for Reinforcement Learning.

```python
import acme
import acme.agents.tf.impala as impala

# Environment
env = bsuite.load_and_wrapper('catch', average_discount=0.99)

# Agent
agent = impala.Impala(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run MPO agent on continuous control tasks

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a maximum-a-posterior policy optimization (MPO) agent. It combines a distributional critic and a stochastic policy.

```python
import acme
import acme.agents.tf.mpo as mpo

# Environment
env = dm_control.make('cheetah-run-v0')

# Agent
agent = mpo.MPO(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run Behaviour Cloning (BC) agent offline (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a behaviour cloning (BC) agent implemented in JAX for offline learning. This agent learns from external data generated by another agent.

```python
import acme
import acme.agents.jax.behavior_cloning as bc

# Environment
env = dm_control.make('cartman-push-v0')

# Agent
agent = bc.BehaviorCloning(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run MCTS agent on Behaviour Suite

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a model-based agent using Monte Carlo Tree Search (MCTS) on the Behaviour Suite. It can use either a simulator or a learned model.

```python
import acme
import acme.agents.tf.mcts as mcts

# Environment
env = bsuite.load_and_wrapper('catch', average_discount=0.99)

# Agent
agent = mcts.MCTS(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run Behaviour Cloning (BC) agent offline

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a behaviour cloning (BC) agent for offline learning. This agent learns from external data generated by another agent.

```python
import acme
import acme.agents.tf.behavior_cloning as bc

# Environment
env = dm_control.make('cartman-push-v0')

# Agent
agent = bc.BehaviorCloning(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Run D4PG agent on DeepMind Control Suite or OpenAI Gym

Source: https://github.com/google-deepmind/acme/blob/master/examples/README.md

Example of a deterministic policy gradient (D4PG) agent. It includes a deterministic policy and a distributional critic. By default, it runs on the 'half cheetah' environment from the OpenAI Gym.

```python
import acme
import acme.agents.tf.d4pg as d4pg

# Environment
env = dm_control.make('cheetah-run-v0')

# Agent
agent = d4pg.D4PG(environment_spec=acme.specs.make_environment_spec(env))

# Replay buffer
replay_buffer = acme.specs.make_replay_buffer(
    agent.builder.replay_buffer_from_config(agent.config.replay_buffer))

# Learner
learner = agent.build_learner(random_key=jnp.array([0, 0]),
                              environment_spec=acme.specs.make_environment_spec(env),
                              replay_buffer=replay_buffer)

# Actor
actor = agent.build_actor(random_key=jnp.array([1, 1]),
                          environment_spec=acme.specs.make_environment_spec(env))

# Environment loop
environment_loop = acme.EnvironmentLoop(env, actor, learner)
environment_loop.run(num_episodes=100)
```

--------------------------------

### Get environment specifications

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Extracts the specifications (shapes, dtypes) of the environment's observations, actions, rewards, and discounts. These specifications are crucial for defining the neural network input/output shapes and the replay buffer's item signature.

```python
environment_spec = specs.make_environment_spec(environment)
```

--------------------------------

### Configure and build a D4PG agent

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Sets up the D4PG algorithm configuration, including learning rate and exploration noise, and then instantiates the D4PGBuilder with this configuration. This prepares the agent for training.

```python
d4pg_config = d4pg.D4PGConfig(learning_rate=3e-4, sigma=0.2)
d4pg_builder = d4pg.D4PGBuilder(d4pg_config)
```

--------------------------------

### Create N-Step Transition Adder

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Sets up an adder for collecting N-step transitions into reverb replay tables. Useful for off-policy learning algorithms.

```python
# Handles preprocessing of data and insertion into replay tables.
adder = reverb_adders.NStepTransitionAdder(
    priority_fns={'priority_table': None},
    client=replay_client,
    n_step=n_step,
    discount=discount)

```

--------------------------------

### Create FeedForwardNetwork for policy and critic

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Prebinds network init methods with dummy observations and actions to simplify parameter initialization. Use to create reusable network objects.

```python
# Create dummy observations and actions to create network parameters.
dummy_action = utils.zeros_like(environment_spec.actions)
dummy_obs = utils.zeros_like(environment_spec.observations)

# Prebind dummy observations and actions so they are not needed in the learner.
policy_network = networks_lib.FeedForwardNetwork(
    init=lambda rng: policy.init(rng, dummy_obs),
    apply=policy.apply)
critic_network = networks_lib.FeedForwardNetwork(
    init=lambda rng: critic.init(rng, dummy_obs, dummy_action),
    apply=critic.apply)
```

--------------------------------

### PPO Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Proximal Policy Optimization (PPO) agent for discrete action spaces using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/ppo/
```

--------------------------------

### DMPO Agent Initialization

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Initializes a Deep Maximum a Posteriori Policy Optimization (DMPO) agent. Networks for policy, critic, and observation processing are passed as arguments.

```python
agent = dmpo.DMPO(
    # Networks defined above.
    policy_network=stochastic_policy_network,
    critic_network=distributional_critic_network,
    # New ResNet visual module, shared by both policy and critic.
    observation_network=shared_resnet,
    # ...
)
```

--------------------------------

### Render and Visualize Agent in Environment

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Wraps the environment to render frames and produce videos of episodes. Used for visualizing the agent's behavior without affecting training data.

```python
# Make the environment render frames and produce videos of episodes.
eval_environment = wrappers.MujocoVideoWrapper(environment, record_every=1)

timestep = eval_environment.reset()

while not timestep.last():
  action = actor.select_action(timestep.observation)
  timestep = eval_environment.step(action)

```

--------------------------------

### Initialize a logger for training results

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Sets up a logger to collect and store training and evaluation data in memory. This logger can later be used for plotting results.

```python
# Specify how to log training data: in this case keeping it in memory.
# NOTE: We create a dict to hold the loggers so we can access the data after
```

--------------------------------

### Initialize TensorFlow Checkpointer and Snapshotter

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Sets up TensorFlow model saving using Checkpointer for restoring with a re-built graph and Snapshotter for internal graph rebuilding. Both require a dictionary of objects to save.

```python
model = snt.Linear(10)
checkpointer = tf2_savers.Checkpointer(objects_to_save={'model': model})
snapshotter = tf2_savers.Snapshotter(objects_to_save={'model': model})
for _ in range(100):
  # ...
  checkpointer.save()
  snapshotter.save()
```

--------------------------------

### Configure Reverb replay table and rate limiter

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Sets up a Reverb replay table with a uniform sampler and FIFO remover, and configures a rate limiter to manage data flow. Use for storing and sampling agent experience.

```python
# Manages the data flow by limiting the sample and insert calls.
rate_limiter = reverb.rate_limiters.SampleToInsertRatio(
    min_size_to_sample=1000,
    samples_per_insert=samples_per_insert,
    error_buffer=2 * batch_size)

# Create a replay table to store previous experience.
replay_tables = [
    reverb.Table(
        name='priority_table',
        sampler=reverb.selectors.Uniform(),
        remover=reverb.selectors.Fifo(),
        max_size=1_000_000,
        rate_limiter=rate_limiter,
        signature=reverb_adders.NStepTransitionAdder.signature(
            environment_spec))
]
```

--------------------------------

### Initialize and Write to CSVLogger

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Logs data to a specified CSV file. Instantiate with a log directory and label, then use the write method to log data.

```python
csv_logger = loggers.CSVLogger(logdir='logged_data', label='my_csv_file')
csv_logger.write({'step': 0, 'reward': 0.0})
```

--------------------------------

### Agent Initialization with Networks

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Agents typically require networks for policy and value functions, passed directly during initialization.

```python
policy_network = ...
critic_network = ...
agent = MyActorCriticAgent(policy_network, critic_network, ...)
```

--------------------------------

### Configure Experiment

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Defines the experiment configuration, including the builder, environment, network, logger, seed, and maximum actor steps.

```python
experiment_config = experiments.ExperimentConfig(
    builder=d4pg_builder,
    environment_factory=make_environment,
    network_factory=network_factory,
    logger_factory=logger_factory,
    seed=0,
    max_num_actor_steps=50_000)  # Each episode is 1000 steps.

```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://github.com/google-deepmind/acme/blob/master/README.md

Use this command to create a Python virtual environment for managing Acme dependencies and avoid version conflicts.

```bash
python3 -m venv acme
source acme/bin/activate
pip install --upgrade pip setuptools wheel
```

--------------------------------

### Initialize and Write to TerminalLogger

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Logs data directly to the terminal. Instantiate with a label and time delta, then use the write method to log data.

```python
terminal_logger = loggers.TerminalLogger(label='TRAINING',time_delta=5)
terminal_logger.write({'step': 0, 'reward': 0.0})
```

--------------------------------

### PPO Agent (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Proximal Policy Optimization agent implemented in JAX. Use for continuous control problems.

```python
acme/agents/jax/ppo/
```

--------------------------------

### Conservative Q-learning (CQL) Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Conservative Q-learning (CQL) agent for offline RL using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/cql/
```

--------------------------------

### Initialize D4PG Learner

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Initializes the Distributed Deep Deterministic Policy Gradient (D4PG) learner with specified network architectures, optimizers, and environment parameters.

```python
key, learner_key = jax.random.split(key)

# The learner updates the parameters (and initializes them).
learner = learning.D4PGLearner(
    policy_network=policy_network,
    critic_network=critic_network,
    random_key=learner_key,
    policy_optimizer=optax.adam(learning_rate),
    critic_optimizer=optax.adam(learning_rate),
    discount=discount,
    target_update_period=target_update_period,
    iterator=dataset,
    # A simple counter object that can periodically sync with a parent counter.
    counter=counting.Counter(parent_counter, prefix='learner', time_delta=0.),
)
```

--------------------------------

### Run Experiment

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Executes the configured experiment, specifying evaluation frequency and number of evaluation episodes.

```python
experiments.run_experiment(
    experiment=experiment_config,
    eval_every=1000,
    num_eval_episodes=1)

```

--------------------------------

### Deep Q-Networks (DQN) Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Deep Q-Networks (DQN) agent for discrete action spaces using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/dqn/
```

--------------------------------

### Load and wrap the environment for D4PG agent

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Loads the 'cartpole_balance' environment from dm_control suite and applies several wrappers: ConcatObservationWrapper to combine observations, CanonicalSpecWrapper to ensure actions are clipped to [-1, 1], and SinglePrecisionWrapper for float32 output compatible with TPUs.

```python
# Control suite environments are dm_env.Environments with additional attributes
# such as a `physics` object, which we use to render the scene.
environment: control.Environment = suite.load('cartpole', 'balance')

# Concatenate the observations (position, velocity, etc).
environment = wrappers.ConcatObservationWrapper(environment)

# Make the environment expect continuous action spec is [-1, 1].
# Note: this is a no-op on dm_control tasks.
environment = wrappers.CanonicalSpecWrapper(environment, clip=True)

# Make the environment output single-precision floats.
# We use this because most TPUs only work with float32.
environment = wrappers.SinglePrecisionWrapper(environment)
```

--------------------------------

### Visualize Training Returns

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Generates a plot of episode returns during training using pandas and matplotlib. Displays the agent's performance over time.

```python
%matplotlib inline
df = pd.DataFrame(env_loop_logger.data)
plt.figure(figsize=(10, 4))
plt.title('Training episodes returns')
plt.xlabel('Training episodes')
plt.ylabel('Episode return')
plt.plot(df['episode_return']);

```

--------------------------------

### Instantiate Environment Wrapper

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Wrappers are used to modify or expose dm_env environments. Additional parameters can control wrapper behavior.

```python
environment = Wrapper(raw_environment, ...)
```

--------------------------------

### Basic Environment Loop in Python

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/overview.md

Illustrates the fundamental interaction loop between an actor and an environment. Assumes environment conforms to the DeepMind Environment API.

```python
while True:
  # Make an initial observation.
  step = environment.reset()
  actor.observe_first(step.observation)

  while not step.last():
    # Evaluate the policy and take a step in the environment.
    action = actor.select_action(step.observation)
    step = environment.step(action)

    # Make an observation and update the actor.
    actor.observe(action, next_step=step)
    actor.update()
```

--------------------------------

### Create Environment Loop

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Initializes an Acme EnvironmentLoop for training. It orchestrates interaction between the environment, actor, and a counter for tracking progress.

```python
env_loop_logger = loggers.InMemoryLogger()

# Create the environment loop used for training.
env_loop = acme.EnvironmentLoop(
    environment,
    actor,
    counter=counting.Counter(parent_counter, prefix='train', time_delta=0.),
    logger=env_loop_logger)

```

--------------------------------

### R2D2 Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Recurrent Replay Distributed DQN (R2D2) agent for discrete action spaces using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/r2d2/
```

--------------------------------

### Initialize Generic Actor

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Initializes a generic actor with a batched feed-forward policy, a variable client for remote updates, and an adder for storing transitions.

```python
key, actor_key = jax.random.split(key)

# A convenience adaptor from FeedForwardPolicy to ActorCore.
actor_core = actor_core_lib.batched_feed_forward_to_actor_core(
    exploration_policy)

# A variable client for updating variables from a remote source.
variable_client = variable_utils.VariableClient(learner, 'policy', device='cpu')
actor = actors.GenericActor(
    actor=actor_core,
    random_key=actor_key,
    variable_client=variable_client,
    adder=adder,
    backend='cpu')

```

--------------------------------

### Behavior Cloning (BC) Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Behavior Cloning (BC) agent for offline RL using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/bc/
```

--------------------------------

### Create a central counter

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Initializes a central counter for synchronizing component counts. Use as a parent counter for other component counters.

```python
parent_counter = counting.Counter(time_delta=0.)
```

--------------------------------

### Behavior Value Estimation (BVE) Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Behavior Value Estimation (BVE) agent for offline RL using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/bve/
```

--------------------------------

### Deep Q-Networks (DQN) Agent in TensorFlow

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Deep Q-Networks (DQN) agent for discrete action spaces using TensorFlow. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/tf/dqn/
```

--------------------------------

### MPO Agent (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Maximum a posteriori Policy Optimisation agent implemented in JAX. Use for continuous control.

```python
acme/agents/jax/mpo/
```

--------------------------------

### IMPALA Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Importance-Weighted Actor-Learner Architectures (IMPALA) agent for discrete action spaces using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/impala/
```

--------------------------------

### Configure D4PG agent hyperparameters

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Sets key hyperparameters for the D4PG agent, including batch size, learning rate, discount factor, n-step transitions, exploration noise, target update period, samples per insert ratio, and critic atom distribution.

```python
key = jax.random.PRNGKey(123)

batch_size = 256
learning_rate = 1e-4
discount = 0.99
n_step = 5  # The D4PG agent learns from n-step transitions.
exploration_sigma = 0.3
target_update_period = 100

# Controls the relative rate of sampled vs inserted items. In this case, items
# are n-step transitions.
samples_per_insert = 32.0

# Atoms used by the categorical distributional critic.
num_atoms = 51
critic_atoms = jnp.linspace(-150., 150., num_atoms)
```

--------------------------------

### Run Training Loop

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Executes the training loop for a specified number of episodes. This function drives the agent's learning process within the environment.

```python
env_loop.run(num_episodes=50)

```

--------------------------------

### N-Step Transition Structure

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Defines the data structure for N-step transitions used in RL replay buffers. For N=1, it includes state, action, reward, discount, next state, and done flag. For N>1, it includes aggregated rewards and discounts.

```default
(s_t, a_t, r_t, d_t, s_{t+1}, e_t)
```

```default
(s_t, a_t, R_{t:t+n}, D_{t:t+n}, s_{t+n}, e_t),
```

--------------------------------

### Create Reverb dataset iterator

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Constructs a TensorFlow dataset from the Reverb server for agent consumption. Use to efficiently pull data from the replay buffer.

```python
# Pull data from the Reverb server into a TF dataset the agent can consume.
dataset = datasets.make_reverb_dataset(
    table='priority_table',
    server_address=replay_client.server_address,
    batch_size=batch_size,
)

# We use multi_device_put here in case this colab is run on a machine with
# multiple accelerator devices, but this works fine with single-device learners
```

--------------------------------

### Episode Adder Trajectory Structure

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Illustrates the data format for storing entire episodes as trajectories using an EpisodeAdder. It includes state, action, reward, discount, and extra information for each step in the episode.

```python
(s_0, a_0, r_0, d_0, e_0,
 s_1, a_1, r_1, d_1, e_1,
          .
          .
          .
 s_T, a_T, r_T, 0., e_T)
```

--------------------------------

### D4PG Agent (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Distributed Distributional Deep Deterministic Policy Gradients agent implemented in JAX. Use for continuous control tasks.

```python
acme/agents/jax/d4pg/
```

--------------------------------

### Import necessary modules for Acme D4PG agent

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Imports a comprehensive set of modules from various libraries including dm_control, IPython, JAX, Haiku, Optax, Reverb, RLax, TensorFlow, and Acme. These are required for building and running the D4PG agent.

```python
%env MUJOCO_GL=egl
from typing import Sequence, Tuple

from dm_control import suite
from dm_control.rl import control
from IPython.display import HTML
import jax
import jax.numpy as jnp
import haiku as hk
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import optax
import reverb
import rlax
import tensorflow as tf

import acme
from acme import specs
from acme import wrappers
from acme.adders import reverb as reverb_adders
from acme.agents.jax import actors
from acme.agents.jax import actor_core as actor_core_lib
from acme.agents.jax.d4pg import learning
from acme.datasets import reverb as datasets
from acme.jax import utils, variable_utils
from acme.jax import networks as networks_lib
from acme.jax.experiments.run_experiment import _disable_insert_blocking, _LearningActor
from acme.utils import counting
from acme.utils import loggers
```

--------------------------------

### MO-MPO Agent (TensorFlow)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Multi-Objective Maximum a posteriori Policy Optimisation agent implemented in TensorFlow. Suitable for multi-objective continuous control.

```python
acme/agents/tf/mompo/
```

--------------------------------

### Critic-Regularized Regression (CRR) Agent in JAX

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Critic-Regularized Regression (CRR) agent for offline RL using JAX. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/jax/crr/
```

--------------------------------

### Behavior Cloning (BC) Agent in TensorFlow

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Behavior Cloning (BC) agent for offline RL using TensorFlow. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/tf/bc/
```

--------------------------------

### Transform functions to Haiku networks

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Transforms simple Python functions into Haiku networks with init and apply methods. Use when defining neural network architectures.

```python
policy = hk.without_apply_rng(hk.transform(policy_fn))
critic = hk.without_apply_rng(hk.transform(critic_fn))
```

--------------------------------

### Define a custom network factory for D4PG

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Creates a factory function to generate D4PG networks (policy and critic) with specified hidden layer sizes. This allows customization of the network architecture based on the environment's specifications.

```python
def network_factory(spec: specs.EnvironmentSpec) -> d4pg.D4PGNetworks:
  return d4pg.make_networks(
      spec,
      # These correspond to sizes of the hidden layers of an MLP.
      policy_layer_sizes=(256, 256),
      critic_layer_sizes=(256, 256),
  )
```

--------------------------------

### Plot Training Results

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Plots the training episodes' returns using pandas and matplotlib. Ensure matplotlib is configured for inline display.

```python
%matplotlib inline
df = pd.DataFrame(logger_dict['evaluator'].data)
plt.figure(figsize=(10, 4))
plt.title('Training episodes returns')
plt.xlabel('Training episodes')
plt.ylabel('Episode return')
plt.plot(df['actor_episodes'], df['episode_return'], label='Training Episodes return')

```

--------------------------------

### Define a custom environment factory for Acme

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Creates a factory function to instantiate a dm_env.Environment, applying wrappers for observation concatenation, canonical action specs, and single-precision floats. This is useful for creating consistent environments for agents.

```python
def make_environment(seed: int) -> dm_env.Environment:
  environment = dm_suite.load('cartpole', 'balance')

  # Make the observations be a flat vector of all concatenated features.
  environment = wrappers.ConcatObservationWrapper(environment)

  # Wrap the environment so the expected continuous action spec is [-1, 1].
  # Note: this is a no-op on 'control' tasks.
  environment = wrappers.CanonicalSpecWrapper(environment, clip=True)

  # Make sure the environment outputs single-precision floats.
  environment = wrappers.SinglePrecisionWrapper(environment)

  return environment
```

--------------------------------

### SAC Agent (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Soft Actor-Critic agent implemented in JAX. Designed for continuous action spaces.

```python
acme/agents/jax/sac/
```

--------------------------------

### DMPO Agent (TensorFlow)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Distributional Maximum a posteriori Policy Optimisation agent implemented in TensorFlow. Use for continuous control.

```python
acme/agents/tf/dmpo/
```

--------------------------------

### TD3 Agent (JAX)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Twin Delayed Deep Deterministic policy gradient agent implemented in JAX. Suitable for continuous control problems.

```python
acme/agents/jax/td3/
```

--------------------------------

### R2D2 Agent in TensorFlow

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Recurrent Replay Distributed DQN (R2D2) agent for discrete action spaces using TensorFlow. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/tf/r2d2/
```

--------------------------------

### Replace Actor with Learning Actor

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Replaces the generic actor with a `_LearningActor` for single-process execution. This actor integrates learning steps with environment interaction to prevent deadlocks.

```python
# NOTE: This is the third of three code cells that are specific to
# single-process execution. (This is done for you when you use an agent
# `Builder` and `run_experiment`.) Everything else is logic shared between the
# two.
actor = _LearningActor(actor, learner, dataset, replay_tables,
                       rate_limiters_max_diff, checkpointer=None)

```

--------------------------------

### Define D4PG deterministic policy network using Haiku

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Defines the deterministic policy network using Haiku. It takes observations, passes them through a LayerNormMLP with two hidden layers of 256 units, and outputs actions scaled to the environment's action spec using a TanhToSpec layer.

```python
# Calculate how big the last layer should be based on total # of actions.
action_spec = environment_spec.actions
action_size = np.prod(action_spec.shape, dtype=int)

# Create the deterministic policy network.
def policy_fn(obs: networks_lib.Observation) -> jnp.ndarray:
  x = obs
  x = networks_lib.LayerNormMLP([256, 256], activate_final=True)(x)
  x = networks_lib.NearZeroInitializedLinear(action_size)(x)
  x = networks_lib.TanhToSpec(action_spec)(x)
  return x

```

--------------------------------

### Define exploration policy with Gaussian noise

Source: https://github.com/google-deepmind/acme/blob/master/examples/tutorial.ipynb

Wraps a policy network to add Gaussian noise for exploration. Use when stochasticity is needed during policy execution.

```python
def exploration_policy(
    params: networks_lib.Params,
    key: networks_lib.PRNGKey,
    observation: networks_lib.Observation,
) -> networks_lib.Action:
  action = policy_network.apply(params, observation)
  if exploration_sigma:
    action = rlax.add_gaussian_noise(key, action, exploration_sigma)
  return action
```

--------------------------------

### Import necessary modules for Acme agent development

Source: https://github.com/google-deepmind/acme/blob/master/examples/quickstart.ipynb

Imports core libraries and specific modules from Acme, TensorFlow, dm_control, and other dependencies required for building RL agents.

```python
from typing import Optional

import collections
from dm_control import suite as dm_suite
import dm_env
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf

from acme import specs
from acme import wrappers
from acme.agents.jax import d4pg
from acme.jax import experiments
from acme.utils import loggers
```

--------------------------------

### MPO Agent (TensorFlow)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Maximum a posteriori Policy Optimisation agent implemented in TensorFlow. Suitable for continuous control tasks.

```python
acme/agents/tf/mpo/
```

--------------------------------

### IMPALA Agent in TensorFlow

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Implementation of the Importance-Weighted Actor-Learner Architectures (IMPALA) agent for discrete action spaces using TensorFlow. Refer to the paper for algorithm details.

```python
https://github.com/deepmind/acme/blob/master/acme/agents/tf/impala/
```

--------------------------------

### D4PG Agent (TensorFlow)

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/agents.md

Distributed Distributional Deep Deterministic Policy Gradients agent implemented in TensorFlow. Use for continuous control tasks.

```python
acme/agents/tf/ddpg/
```

--------------------------------

### Sequence Adder Structure

Source: https://github.com/google-deepmind/acme/blob/master/docs/user/components.md

Defines the data structure for storing fixed-length sequences using a SequenceAdder. It includes state, action, reward, discount, and extra information for a sequence of length 'n'. Sequences can be overlapping or non-overlapping.

```python
(s_0, a_0, r_0, d_0, e_0,
 s_1, a_1, r_1, d_1, e_1,
          .
          .
          .
 s_n, a_n, r_n, d_n, e_n)
```