### Starting Reverb Server with Config (Bash)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Provides an example of starting the `reverb_server` script from the command line using a textproto configuration string. This configures the server port and defines a table with specific sampler, remover, size, and rate limiter settings.

```bash
$ reverb_server --config="
port: 8000
tables: {
  table_name: \"my_table\"
  sampler: {
    fifo: true
  }
  remover: {
    fifo: true
  }
  max_size: 200 max_times_sampled: 5
  rate_limiter: {
    min_size_to_sample: 1
    samples_per_insert: 1
    min_diff: $(python3 -c \"import sys; print(-sys.float_info.max)\")
    max_diff: $(python3 -c \"import sys; print(sys.float_info.max)\")
  }
}"
```

--------------------------------

### Install Reverb Dependencies - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Commands to install necessary Python packages: TensorFlow, dm-tree, and dm-reverb using pip, required for running the frame stacking examples.

```python
!pip install tf
!pip install dm-tree
!pip install dm-reverb
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Installs required Python packages: numpy, dm-tree, and dm-reverb with tensorflow support, using pip commands executed within the environment.

```Python
!pip install numpy --upgrade
!pip install dm-tree
!pip install dm-reverb[tensorflow]
```

--------------------------------

### Run and Enter Reverb Docker Container

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Runs the previously built Docker container, mounting the Reverb source directory and the user's git configuration. It starts an interactive session inside the container.

```shell
docker run --rm -it \
  --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
  --mount "type=bind,src=$HOME/.gitconfig,dst=/etc/gitconfig,ro" \
  --name reverb tensorflow:reverb bash
```

--------------------------------

### Start Reverb Server (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Initializes and starts a Reverb server instance. It configures a table named 'my_table' with a Uniform sampler, Fifo remover, a maximum size of 100, and a minimum size rate limiter of 1.

```Python
import reverb

server = reverb.Server(tables=[
    reverb.Table(
        name='my_table',
        sampler=reverb.selectors.Uniform(),
        remover=reverb.selectors.Fifo(),
        max_size=100,
        rate_limiter=reverb.rate_limiters.MinSize(1)),
    ],
)
```

--------------------------------

### Install Reverb via pip (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Installs the Reverb library using pip. The first command includes TensorFlow as a dependency, ensuring version compatibility. The second command installs Reverb without checking or installing TensorFlow.

```Shell
$ pip install dm-reverb[tensorflow]
```

```Shell
$ pip install dm-reverb
```

--------------------------------

### Install Reverb Python Wheel

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Installs the generated Reverb Python wheel file using pip. The command uses the `$PYTHON_BIN_PATH` environment variable to ensure the correct Python interpreter is used.

```shell
$PYTHON_BIN_PATH -mpip install --upgrade /tmp/reverb_build/dist/*.whl
```

--------------------------------

### Install Reverb Nightly via pip (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Installs the nightly build of the Reverb library using pip. The first command includes TensorFlow as a dependency, ensuring version compatibility. The second command installs Reverb without checking or installing TensorFlow.

```Shell
$ pip install dm-reverb-nightly[tensorflow]
```

```Shell
$ pip install dm-reverb-nightly
```

--------------------------------

### Example: Store Stacked Frames (Stride < Stack Size) - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

A commented-out example illustrating how to call `store_stacked` with `stride` less than `stack_size`, explaining that this configuration results in overlapping stacks being stored, potentially doubling storage before compression.

```python
# Create trajectories with 4 frames stacked together, 2 frames shared between
# stacks and create sequences of 3 stacks. Note that since we stack the frames
# BEFORE sending it to Reverb, most stacks will be stored twice resulting in
# double the storage (before compression is applied).
#
# For example, the first 12 steps will result in the following 3 samplable
# items:
#
#   [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
#

```

--------------------------------

### Example: Store Stacked Frames (Stride = Stack Size) - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Demonstrates calling the `store_stacked` function with parameters that result in non-overlapping stacks (`stride` equals `stack_size`), showing how sequences of 3 stacks are formed from the generated frames.

```python
# Create trajectories with 4 frames stacked together, no frames shared
# between stacks and create sequences of 3 stacks. For example, the first 16
# steps will result in the following 2 samplable items:
#
#   [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
#
#     -> [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
#     -> [[5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]]
#

store_stacked(stack_size=4, stride=4, sequence_length=3)
```

--------------------------------

### Setting up Reverb Server and Client for Complete Episodes

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a reverb.Server with a single table configured for complete episodes, including prioritized sampling, FIFO removal, rate limiting, and a detailed signature. Also initializes a reverb.Client connected to the server.

```python
EPISODE_LENGTH = 150

complete_episode_server = reverb.Server(tables=[
    reverb.Table(
        name='my_table',
        sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
        remover=reverb.selectors.Fifo(),
        max_size=int(1e6),
        # Sets Rate Limiter to a low number for the examples.
        # Read the Rate Limiters section for usage info.
        rate_limiter=reverb.rate_limiters.MinSize(2),
        # The signature is optional but it is good practice to set it as it
        # enables data validation and easier dataset construction. Note that
        # the number of observations is larger than the number of actions.
        # The extra observation is the terminal state where no action is
        # taken.
        signature={
            'actions':
                tf.TensorSpec([EPISODE_LENGTH, *ACTION_SPEC.shape],
                              ACTION_SPEC.dtype),
            'observations':
                tf.TensorSpec([EPISODE_LENGTH + 1, *OBSERVATION_SPEC.shape],
                              OBSERVATION_SPEC.dtype),
        },
    ),
])

# Initializes the reverb client on the same port.
client = reverb.Client(f'localhost:{complete_episode_server.port}')
```

--------------------------------

### Initialize Reverb Server and Client

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Creates a Reverb server with a single table configured with specific sampler, remover, rate limiter, and signature. Initializes a client connected to the server's port.

```Python
simple_server = reverb.Server(
    tables=[
        reverb.Table(
            name='my_table',
            sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
            remover=reverb.selectors.Fifo(),
            max_size=int(1e6),
            # Sets Rate Limiter to a low number for the examples.
            # Read the Rate Limiters section for usage info.
            rate_limiter=reverb.rate_limiters.MinSize(2),
            # The signature is optional but it is good practice to set it as it
            # enables data validation and easier dataset construction. Note that
            # we prefix all shapes with a 3 as the trajectories we'll be writing
            # consist of 3 timesteps.
            signature={
                'actions':
                    tf.TensorSpec([3, *ACTION_SPEC.shape], ACTION_SPEC.dtype),
                'observations':
                    tf.TensorSpec([3, *OBSERVATION_SPEC.shape],
                                  OBSERVATION_SPEC.dtype),
            },
        )
    ],
    # Sets the port to None to make the server pick one automatically.
    # This can be omitted as it's the default.
    port=None)

# Initializes the reverb client on the same port as the server.
client = reverb.Client(f'localhost:{simple_server.port}')
```

--------------------------------

### Creating a Reverb Server with Multiple Priority Tables (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a Reverb server instance configured with two priority tables, 'my_table_a' and 'my_table_b'. Each table uses a Prioritized sampler, a FIFO remover, a maximum size of 1 million, and a MinSize rate limiter set to 1 for demonstration. It also creates a client connected to this server.

```python
multitable_server = reverb.Server(
    tables=[
        reverb.Table(
            name='my_table_a',
            sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
            remover=reverb.selectors.Fifo(),
            max_size=int(1e6),
            # Sets Rate Limiter to a low number for the examples.
            # Read the Rate Limiters section for usage info.
            rate_limiter=reverb.rate_limiters.MinSize(1)),
        reverb.Table(
            name='my_table_b',
            sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
            remover=reverb.selectors.Fifo(),
            max_size=int(1e6),
            # Sets Rate Limiter to a low number for the examples.
            # Read the Rate Limiters section for usage info.
            rate_limiter=reverb.rate_limiters.MinSize(1)),
    ])

client = reverb.Client('localhost:{}'.format(multitable_server.port))
```

--------------------------------

### Define Dummy RL Environment Steps

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Defines TensorFlow specs for observations and actions, and creates dummy functions for agent and environment steps for demonstration purposes.

```Python
OBSERVATION_SPEC = tf.TensorSpec([10, 10], tf.uint8)
ACTION_SPEC = tf.TensorSpec([2], tf.float32)

def agent_step(unused_timestep) -> tf.Tensor:
  return tf.cast(tf.random.uniform(ACTION_SPEC.shape) > .5,
                 ACTION_SPEC.dtype)

def environment_step(unused_action) -> tf.Tensor:
  return tf.cast(tf.random.uniform(OBSERVATION_SPEC.shape, maxval=256),
                 OBSERVATION_SPEC.dtype)
```

--------------------------------

### Creating Reverb Server with Prioritized Sampler and FIFO Remover (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a Reverb server with a single table named 'my_table'. This table is configured to use a Prioritized sampler, a FIFO remover, a maximum size of 1 million, and a MinSize rate limiter set to 100.

```python
reverb.Server(tables=[
    reverb.Table(
        name='my_table',
        sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
        remover=reverb.selectors.Fifo(),
        max_size=int(1e6),
        rate_limiter=reverb.rate_limiters.MinSize(100)),
])
```

--------------------------------

### Restoring Reverb Server from Checkpoint (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Shows how to initialize a Reverb server instance by loading data from a previously saved checkpoint. It requires specifying the checkpoint root directory and the table configurations used by the original server.

```python
# The checkpointer accepts the path of the root directory in which checkpoints
# are written. If we pass the root directory of the checkpoints written above
# then the new server will load the most recent checkpoint written from the old
# server.
checkpointer = reverb.platform.checkpointers_lib.DefaultCheckpointer(
  path=checkpoint_path.rsplit('/', 1)[0])

# The arguments passed to `tables=` must be the same as those used by the
# `Server` that wrote the checkpoint.
server = reverb.Server(tables=[...], checkpointer=checkpointer)
```

--------------------------------

### Creating Reverb Server with MaxHeap Sampler and MinHeap Remover (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a Reverb server with a single table named 'my_priority_queue'. This table is configured as a max priority queue using a MaxHeap sampler and a MinHeap remover, with `max_times_sampled=1` to remove items after sampling. It has a maximum size of 1000 and a MinSize rate limiter.

```python
max_size = 1000
reverb.Server(tables=[
    reverb.Table(
        name='my_priority_queue',
        sampler=reverb.selectors.MaxHeap(),
        remover=reverb.selectors.MinHeap(),
        max_size=max_size,
        rate_limiter=reverb.rate_limiters.MinSize(int(0.95 * max_size)),
        max_times_sampled=1,
    )
])
```

--------------------------------

### Import Reverb and TensorFlow

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Imports the necessary libraries, reverb and tensorflow, required for building and interacting with the Reverb server and client.

```Python
import reverb
import tensorflow as tf
```

--------------------------------

### Creating Reverb Server with Queue and Circular Buffer Tables (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a Reverb server with two tables: 'my_queue' created using the `Table.queue` helper for queue behavior, and 'my_circular_buffer' configured with FIFO sampler/remover and `max_times_sampled=1` to simulate a circular buffer. Both have a max size of 10000.

```python
reverb.Server(
    tables=[
        reverb.Table.queue(name='my_queue', max_size=10000),
        reverb.Table(
            name='my_circular_buffer',
            sampler=reverb.selectors.Fifo(),
            remover=reverb.selectors.Fifo(),
            max_size=10000,
            max_times_sampled=1,
            rate_limiter=reverb.rate_limiters.MinSize(1)),
    ])
```

--------------------------------

### Create Reverb Client and Check Server Info (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Creates a client instance to connect to the running Reverb server on localhost using its assigned port. It then prints information about the connected server.

```Python
client = reverb.Client(f'localhost:{server.port}')
print(client.server_info())
```

--------------------------------

### Build Reverb Python Wheel

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Runs the Bazel-built script to package the compiled Reverb code into a Python wheel file (.whl). The output is directed to the specified destination directory.

```shell
./bazel-bin/reverb/pip_package/build_pip_package \
  --dst /tmp/reverb_build/dist/
```

--------------------------------

### License Header

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Standard Apache 2.0 license header for the file, specifying terms of use and distribution.

```Python
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

--------------------------------

### Apache License Header - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Standard Apache 2.0 license header for the file, specifying the terms under which the code is distributed.

```python
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

--------------------------------

### Creating Reverb Server with SampleToInsertRatio Rate Limiter (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Initializes a Reverb server with a single table named 'my_table'. It uses a Prioritized sampler, FIFO remover, max size of 1 million, and a SampleToInsertRatio rate limiter configured with `samples_per_insert=3.0`, `min_size_to_sample=3`, and `error_buffer=3.0`. This limiter is intended for distributed environments.

```python
reverb.Server(
    tables=[
        reverb.Table(
            name='my_table',
            sampler=reverb.selectors.Prioritized(priority_exponent=0.8),
            remover=reverb.selectors.Fifo(),
            max_size=int(1e6),
            rate_limiter=reverb.rate_limiters.SampleToInsertRatio(
                samples_per_insert=3.0, min_size_to_sample=3,
                error_buffer=3.0)),
    ])
```

--------------------------------

### Writing Complete Episodes to Reverb

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Demonstrates writing multiple complete episodes to the Reverb server using a trajectory_writer. It appends individual steps, handles the final observation, creates an item spanning the entire episode, and ends the episode, blocking until insertion is confirmed.

```python
# Writes whole episodes of varying length to a Reverb server.

NUM_EPISODES = 10

# We know that episodes are at most 150 steps so we set the writer buffer size
# to 151 (to capture the final observation).
with client.trajectory_writer(num_keep_alive_refs=151) as writer:
  for _ in range(NUM_EPISODES):
    timestep = environment_step(None)

    for _ in range(EPISODE_LENGTH):
      action = agent_step(timestep)
      writer.append({'action': action, 'observation': timestep})

      timestep = environment_step(action)

    # The astute reader will recognize that the final timestep has not been
    # appended to the writer. We'll go ahead and add it WITHOUT an action. The
    # writer will automatically fill in the gap with `None` for the action
    # column.
    writer.append({'observation': timestep})

    # Now that the entire episode has been added to the writer buffer we can an
    # item with a trajectory that spans the entire episode. Note that the final
    # action must not be included as it is None and the trajectory would be
    # rejected if we tried to include it.
    writer.create_item(
        table='my_table',
        priority=1.5,
        trajectory={
            'actions': writer.history['action'][:-1],
            'observations': writer.history['observation'][:],
        })

    # This call blocks until all the items (in this case only one) have been
    # sent to the server, inserted into respective tables and confirmations
    # received by the writer.
    writer.end_episode(timeout_ms=1000)

    # Ending the episode also clears the history property which is why we are
    # able to use `[:]` in when defining the trajectory above.
    assert len(writer.history['action']) == 0
    assert len(writer.history['observation']) == 0
```

--------------------------------

### Define function to store flat and stack on sample (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Defines the `store_flat` function which demonstrates storing data as flat sequences in Reverb and performing the frame stacking operation after the data is sampled. It sets up a Reverb server and client, writes data using a trajectory writer, creates a `TrajectoryDataset` to sample the flat data, and defines a mapping function `_stack` to transform the flat samples into stacked sequences. Requires `reverb` and `tensorflow`.

```python
def store_flat(stack_size: int, sequence_length: int):
  """Simple example where frames are sent to Reverb and stacked after sampled.

  Args:
    stack_size: The number of frames to stack.
    sequence_length: The number of stacks in each sampleable item.
  """
  server = reverb.Server([reverb.Table.queue('flat_frames', 100)])
  client = server.localhost_client()

  # Insert flat sequences that can be stacked into the desired shape after
  # sampling.
  flat_sequence_length = sequence_length + stack_size - 1
  with client.trajectory_writer(flat_sequence_length) as writer:
    for i, frame in enumerate(frame_generator(flat_sequence_length * 5)):
      writer.append(frame)

      if i + 1 >= flat_sequence_length:
        writer.create_item(table='flat_frames',
                           trajectory=writer.history[-flat_sequence_length:],
                           priority=1.0)

  # Create a dataset that samples sequences of flat frames.
  flat_dataset = reverb.TrajectoryDataset(
      server_address=client.server_address,
      table='flat_frames',
      max_in_flight_samples_per_worker=2,
      dtypes=tf.as_dtype(FRAME_DTYPE),
      shapes=tf.TensorShape((flat_sequence_length,) + FRAME_SHAPE),
      max_samples=2)

  # Create a transformation that stacks the frames.
  def _stack(sample):
    stacks = []
    for i in range(sequence_length):
      stacks.append(sample.data[i:i+stack_size])
    return reverb.ReplaySample(
        info=sample.info,
        data=tf.stack(stacks))

  stacked_dataset = flat_dataset.map(_stack)


  # Print the result.
  for sequence in stacked_dataset:
    print(sequence.data)
```

--------------------------------

### Import Required Libraries - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Imports necessary modules for frame stacking and Reverb usage, including deque for buffering, numpy for array manipulation, reverb for data storage, and tensorflow for dataset creation.

```python
from collections import deque

import numpy as np
import reverb
import tensorflow as tf
```

--------------------------------

### Sample Items (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Samples a specified number of items from the 'my_table' on the Reverb server. The `client.sample()` method returns a generator, which is converted to a list for printing.

```Python
# client.sample() returns a generator.
print(list(client.sample('my_table', num_samples=2)))
```

--------------------------------

### Inserting Sequences into Multiple Reverb Tables (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Demonstrates how to use a `trajectory_writer` to append data steps and create items in different tables ('my_table_a' and 'my_table_b') based on the step number. It shows creating items with varying trajectory lengths and priorities.

```python
with client.trajectory_writer(num_keep_alive_refs=3) as writer:
  timestep = environment_step(None)

  for step in range(4):
    writer.append({'timestep': timestep})

    action = agent_step(timestep)
    timestep = environment_step(action)

    if step >= 1:
      writer.create_item(
          table='my_table_b',
          priority=4-step,
          trajectory=writer.history['timestep'][-2:])

    if step >= 2:
      writer.create_item(
          table='my_table_a',
          priority=4-step,
          trajectory=writer.history['timestep'][-3:])
```

--------------------------------

### Creating a Reverb Checkpoint (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Demonstrates how to trigger a checkpoint operation on a Reverb server using the client. The method returns the path where the checkpoint was saved.

```python
# client.checkpoint() returns the path the checkpoint was written to.
checkpoint_path = client.checkpoint()
```

--------------------------------

### Sampling and Batching Complete Episodes (TensorFlow)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Creates a reverb.TrajectoryDataset from the server table signature, configured to sample complete episodes. It then batches the dataset into groups of 128 episodes and iterates through the first batch, printing the shapes of the sample info and data, showing the expected dimensions for batched episodes.

```python
# Each sample is an entire episode.
# Adjusts the expected shapes to account for the whole episode length.
dataset = reverb.TrajectoryDataset.from_table_signature(
  server_address=f'localhost:{complete_episode_server.port}',
  table='my_table',
  max_in_flight_samples_per_worker=10,
  rate_limiter_timeout_ms=10)

# Batches 128 episodes together.
# Each item is an episode of the format (observations, actions) as above.
# Shape of items are now ([128, 151, 10, 10], [128, 150, 2]).
dataset = dataset.batch(128)

# Sample has type reverb.ReplaySample.
for sample in dataset.take(1):
  # Results in the following format.
  print(sample.info.key)              # ([128], uint64)
  print(sample.info.probability)      # ([128], float64)

  print(sample.data['observations'])  # ([128, 151, 10, 10], uint8)
  print(sample.data['actions'])       # ([128, 150, 2], float32)
```

--------------------------------

### Batching and Sampling Streaming Dataset

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Batches the streaming dataset into groups of 2 sequences. Iterates through the first batch and prints the shapes of the sample info (key, probability) and data (observations, actions).

```python
# Batches 2 sequences together.
# Shapes of items is now [2, 3, 10, 10].
batched_dataset = dataset.batch(2)

for sample in batched_dataset.take(1):
  # Results in the following format.
  print(sample.info.key)              # ([2], uint64)
  print(sample.info.probability)      # ([2], float64)

  print(sample.data['observations'])  # ([2, 3, 10, 10], uint8)
  print(sample.data['actions'])       # ([2, 3, 2], float32)
```

--------------------------------

### Compile Reverb inside Docker

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Executes the configuration script and compiles Reverb using Bazel inside the Docker container. This prepares the build artifacts for creating the Python wheel.

```shell
python3.9 configure.py
bazel build -c opt //reverb/pip_package:build_pip_package
```

--------------------------------

### Configuring Prioritized Experience Replay Table (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

This snippet demonstrates configuring a Reverb Table for Prioritized Experience Replay. It uses a Prioritized sampler with an exponent of 0.8, a FIFO remover, a maximum size of 1000 items, and requires at least 100 items before sampling.

```python
reverb.Table(
     name='my_prioritized_experience_replay_buffer',
     sampler=reverb.selectors.Prioritized(0.8),
     remover=reverb.selectors.Fifo(),
     max_size=1000,
     rate_limiter=reverb.rate_limiters.MinSize(100),
)
```

--------------------------------

### Configuring Uniform Experience Replay Table (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

This snippet shows how to configure a Reverb Table for Uniform Experience Replay. It uses a Uniform sampler, a FIFO remover, has a maximum size of 1000 items, and requires at least 100 items before sampling is allowed.

```python
reverb.Table(
     name='my_uniform_experience_replay_buffer',
     sampler=reverb.selectors.Uniform(),
     remover=reverb.selectors.Fifo(),
     max_size=1000,
     rate_limiter=reverb.rate_limiters.MinSize(100),
)
```

--------------------------------

### Build Reverb Release Docker Container (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Builds the Docker container image required for creating Reverb releases. It pulls the base image, disables caching, tags the image, and specifies the target TensorFlow version and Python versions as build arguments.

```shell
docker build --pull --no-cache \
  --tag tensorflow:reverb_release \
  --build-arg tensorflow_pip=tensorflow~=2.14.0 \
  --build-arg python_version="python3.9 python3.10 python3.11" \
  - < "$REVERB_DIR/docker/release.dockerfile"
```

--------------------------------

### Build Reverb Package (Stable TensorFlow) (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Executes the Reverb build process inside the Docker container to create packages against a stable TensorFlow release (e.g., tensorflow~=2.14.0). It mounts the local repository, runs the build script with cleaning and cache clearing enabled, specifies the TensorFlow dependency override, and targets multiple Python versions.

```shell
docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
  tensorflow:reverb_release bash oss_build.sh --clean true \
  --clear_bazel_cache true --tf_dep_override "tensorflow~=2.14.0" \
  --release --python "3.9 3.10 3.11"
```

--------------------------------

### Creating a Reverb TrajectoryDataset (Streaming)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Creates a reverb.TrajectoryDataset from a server table signature, configured for streaming large sequences that may not fit in memory. Sets max_in_flight_samples_per_worker to control concurrency.

```python
dataset = reverb.TrajectoryDataset.from_table_signature(
  server_address=f'localhost:{simple_server.port}',
  table='my_table',
  max_in_flight_samples_per_worker=10)
```

--------------------------------

### Build Reverb Debug Package (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Executes the Reverb build process inside the Docker container to create a debug version of the package. Debug builds are significantly larger and are placed in a dedicated subdirectory (`./dist/debug/`).

```shell
docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
  tensorflow:reverb_release bash oss_build.sh --clean true --debug_build true \
  --clear_bazel_cache true --output_dir /tmp/reverb/dist/debug/ \
  --tf_dep_override "tensorflow~=2.14.0" --release --python "3.9 3.10 3.11"
```

--------------------------------

### Configuring Queue Table (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

This snippet shows two ways to configure a Reverb Table as a Queue. Items are sampled and removed using FIFO strategy, limited to 1000 items, and each item can be sampled only once. The rate limiter ensures inserts block when full and samples block when empty.

```python
reverb.Table(
    name='my_queue',
    sampler=reverb.selectors.Fifo(),
    remover=reverb.selectors.Fifo(),
    max_size=1000,
    max_times_sampled=1,
    rate_limiter=reverb.rate_limiters.Queue(size=1000),
)
```

```python
# Or use the helper classmethod `.queue`.
reverb.Table.queue(name='my_queue', max_size=1000)
```

--------------------------------

### Set Reverb Directory Variable (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Sets a local shell variable `REVERB_DIR` to the path of your local Reverb GitHub repository. This variable is used in subsequent commands to reference the repository location.

```shell
export REVERB_DIR=/path/to/reverb/github/repo
```

--------------------------------

### Call store_flat with stack_size 2, sequence_length 3 (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Calls the `store_flat` function to demonstrate the process of storing flat sequences and stacking them during sampling. It configures the process to create samples consisting of 3 stacks, where each stack contains 2 frames. This method is more memory and network efficient when there is significant overlap between desired stacks.

```python
store_flat(stack_size=2, sequence_length=3)
```

--------------------------------

### Store Stacked Frames in Reverb - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Implements a function to stack frames using a deque buffer and write sequences of these stacks to a Reverb queue table. It demonstrates appending stacked frames, creating sampleable items, and creating a Reverb dataset to sample the stored sequences.

```python
def store_stacked(stack_size: int, stride: int, sequence_length: int):
  """Simple example where frames are stacked before sent to Reverb.

  If `stride` < `stack_size` then stacks will "overlap".
  If `stride` == `stack_size` then stacks will be adjecent.
  If `stride` > `stack_size` then frames between stacks will be dropped.

  Args:
    stack_size: The number of frames to stack.
    stride: The number of frames between each stack is created.
    sequence_length: The number of stacks in each sampleable item.
  """
  server = reverb.Server([reverb.Table.queue('stacked_frames', 100)])
  client = server.localhost_client()

  with client.trajectory_writer(sequence_length) as writer:
    # Create a circular buffer of the `stack_size` most recent frames.
    buffer = deque(maxlen=stack_size)

    for i, frame in enumerate(frame_generator(5 * stride * sequence_length)):
      buffer.append(frame)

      # We can't insert anything before the first stack is full.
      if len(buffer) < stack_size or (i + 1) % stride != 0:
        continue

      # Stack the frames in buffer and insert the data into Reverb. The shape of
      # the stack is [stack_size, width, height].
      writer.append(np.stack(buffer))

      # If `sequence_length` full stacks have been written then insert an item
      # that can be sampled.
      stacks_written = (i + 1) // stride - (stack_size - 1) // stride
      if stacks_written >= sequence_length:
        writer.create_item(table='stacked_frames',
                           trajectory=writer.history[-sequence_length:],
                           priority=1.0)

  # Create a dataset that samples sequences of stacked frames.
  dataset = reverb.TrajectoryDataset(
      server_address=client.server_address,
      table='stacked_frames',
      max_in_flight_samples_per_worker=2,
      dtypes=tf.as_dtype(FRAME_DTYPE),
      shapes=tf.TensorShape((sequence_length, stack_size) + FRAME_SHAPE),
      max_samples=2)

  # Print the result.
  for sequence in dataset.take(2):
    print(sequence.data)
```

--------------------------------

### Build Reverb Docker Container

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Builds the Docker container for Reverb development. By default, it uses Python 3.9, but can be configured for Python 3.10 or 3.11 using the `python_version` build argument. Requires the Reverb source directory.

```shell
docker build --tag tensorflow:reverb - < "$REVERB_DIR/docker/release.dockerfile"
```

```shell
docker build --tag tensorflow:reverb \
    --build-arg python_version=python3.10 \
    - < "$REVERB_DIR/docker/release.dockerfile"
```

--------------------------------

### Insert Overlapping Trajectories into Reverb Table

Source: https://github.com/google-deepmind/reverb/blob/master/examples/demo.ipynb

Uses a Reverb client writer to simulate agent-environment interaction, appending steps and creating items consisting of overlapping 3-step trajectories in 'my_table'.

```Python
# Dynamically adds trajectories of length 3 to 'my_table' using a client writer.

with client.trajectory_writer(num_keep_alive_refs=3) as writer:
  timestep = environment_step(None)
  for step in range(4):
    action = agent_step(timestep)
    writer.append({'action': action, 'observation': timestep})
    timestep = environment_step(action)

    if step >= 2:
      # In this example, the item consists of the 3 most recent timesteps that
      # were added to the writer and has a priority of 1.5.
      writer.create_item(
          table='my_table',
          priority=1.5,
          trajectory={
              'actions': writer.history['action'][-3:],
              'observations': writer.history['observation'][-3:],
          }
      )
```

--------------------------------

### Citing the Reverb Paper (BibTeX)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Provides the recommended BibTeX entry for citing the Reverb paper in academic work.

```bibtex
@misc{cassirer2021reverb,
      title={Reverb: A Framework For Experience Replay},
      author={Albin Cassirer and Gabriel Barth-Maron and Eugene Brevdo and Sabela Ramos and Toby Boyd and Thibault Sottiaux and Manuel Kroiss},
      year={2021},
      eprint={2102.04736},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```

--------------------------------

### Build Reverb Package (RC TensorFlow) (Shell)

Source: https://github.com/google-deepmind/reverb/blob/master/reverb/pip_package/README.md

Executes the Reverb build process inside the Docker container to create packages against a TensorFlow Release Candidate (e.g., tensorflow==2.14.0rc0). It requires a strict dependency override for RC builds due to pip version comparison behavior.

```shell
docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
  tensorflow:reverb_release bash oss_build.sh --clean true \
  --clear_bazel_cache true --tf_dep_override "tensorflow==2.14.0rc0" \
  --release --python "3.9 3.10 3.11"
```

--------------------------------

### Call store_stacked with stack_size 2, stride 3, sequence_length 3 (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Calls the `store_stacked` function with parameters that result in non-overlapping stacks and potential frame dropping. It sets the stack size to 2, the stride to 3, and the sequence length to 3. This configuration is less efficient if frames need to be reused.

```python
store_stacked(stack_size=2, stride=3, sequence_length=3)
```

--------------------------------

### Call store_stacked with stack_size 4, stride 2, sequence_length 3 (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Calls the `store_stacked` function to configure data storage and sampling in Reverb. This specific call sets the stack size to 4, the stride between stacks to 2, and the length of the sampled sequence (number of stacks) to 3. This configuration results in overlapping stacks and sequences.

```python
store_stacked(stack_size=4, stride=2, sequence_length=3)
```

--------------------------------

### Define Frame Shape and Generator - Python

Source: https://github.com/google-deepmind/reverb/blob/master/examples/frame_stacking.ipynb

Defines the shape and data type for individual frames and provides a simple generator function that yields frames with increasing integer values for demonstration purposes.

```python
FRAME_SHAPE = (16, 16)  # [width, height]
FRAME_DTYPE = np.uint8


def frame_generator(max_num_frames: int = 1000):
  for i in range(1, max_num_frames + 1):
    yield np.ones(FRAME_SHAPE, dtype=FRAME_DTYPE) * i
```

--------------------------------

### Insert Trajectory (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Uses a `trajectory_writer` to append multiple data elements (`{'a': ..., 'b': ...}`) and then creates a single item that references these elements as a trajectory in 'my_table'. The writer is flushed to ensure the item is inserted.

```Python
# Appends three data elements and inserts a single item which references all
# of them as {'a': [2, 3, 4], 'b': [12, 13, 14]}.
with client.trajectory_writer(num_keep_alive_refs=3) as writer:
  writer.append({'a': 2, 'b': 12})
  writer.append({'a': 3, 'b': 13})
  writer.append({'a': 4, 'b': 14})

  # Create an item referencing all the data.
  writer.create_item(
      table='my_table',
      priority=1.0,
      trajectory={
          'a': writer.history['a'][:],
          'b': writer.history['b'][:],
      })

  # Block until the item has been inserted and confirmed by the server.
  writer.flush()
```

--------------------------------

### Insert Single Item (Python)

Source: https://github.com/google-deepmind/reverb/blob/master/README.md

Inserts a single data element `[0, 1]` as an item into the 'my_table' on the Reverb server with a priority of 1.0.

```Python
client.insert([0, 1], priorities={'my_table': 1.0})
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.