WeaviateEx Batch RetryQueue

https://hexdocs.pm/weaviate_ex/0.7.4/
Admin
WeaviateEx Batch RetryQueue manages automatic re-queuing of failed batch objects by tracking failed...
Tokens:3,185
Snippets:35
Trust Score:9
Update:3 months ago
Show doc for...
Tokens:
Raw
### RetryQueue Start and Usage

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Demonstrates how to start the RetryQueue GenServer and use its core functions for enqueuing failed objects, checking retry counts, draining the queue, and clearing it.

```APIDOC
## RetryQueue.start_link/1

### Description
Starts the retry queue GenServer.

### Function Signature
`start_link(opts)`

### Parameters
#### Keyword Arguments
- **client** (Weaviate.Client) - Required - The Weaviate client instance.
- **max_retries** (integer) - Required - The maximum number of retries for each object.
- **base_delay_ms** (integer) - Required - The base delay in milliseconds for exponential backoff.
- **on_permanent_failure** (function) - Required - A callback function to handle objects that exceed the maximum retry limit.

## RetryQueue.enqueue_failed/2

### Description
Enqueues failed objects for retry.

### Function Signature
`enqueue_failed(pid, failed_objects)`

### Parameters
#### Path Parameters
- **pid** (pid) - Required - The PID of the RetryQueue GenServer.
- **failed_objects** (list) - Required - A list of failed objects to enqueue.

## RetryQueue.get_retry_count/2

### Description
Gets the retry count for a specific UUID.

### Function Signature
`get_retry_count(pid, uuid)`

### Parameters
#### Path Parameters
- **pid** (pid) - Required - The PID of the RetryQueue GenServer.
- **uuid** (string) - Required - The UUID of the object to check.

## RetryQueue.drain/1

### Description
Drains all queued objects for manual processing.

### Function Signature
`drain(pid)`

### Parameters
#### Path Parameters
- **pid** (pid) - Required - The PID of the RetryQueue GenServer.

## RetryQueue.clear/1

### Description
Clears all queued objects and resets retry counts.

### Function Signature
`clear(pid)`

### Parameters
#### Path Parameters
- **pid** (pid) - Required - The PID of the RetryQueue GenServer.
```

--------------------------------

### Start and Use Retry Queue

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Starts the retry queue GenServer and demonstrates enqueuing failed objects, checking retry counts, draining the queue, and clearing it.

```elixir
client = :fake_client
failed_objects = ["uuid-123", "uuid-456"]

# Start the retry queue
{:ok, pid} = RetryQueue.start_link(
  client: client,
  max_retries: 3,
  base_delay_ms: 1000,
  on_permanent_failure: fn objects ->
    Logger.error("Permanent failures: #{length(objects)}")
  end
)

# Enqueue failed objects for retry
:ok = RetryQueue.enqueue_failed(pid, failed_objects)

# Get retry count for a specific UUID
count = RetryQueue.get_retry_count(pid, "uuid-123")

# Drain all queued objects (for manual processing)
{:ok, objects} = RetryQueue.drain(pid)

# Clear the queue
:ok = RetryQueue.clear(pid)
```

--------------------------------

### start_link

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Starts the retry queue GenServer process with configurable options for client, maximum retries, delay, and failure callbacks.

```APIDOC
## start_link(opts)

### Description
Start the retry queue GenServer.

### Parameters
#### Path Parameters
- **opts** (keyword()) - Required - Options for starting the GenServer.
  * **`:client`** - WeaviateEx client (required)
  * **`:max_retries`** - Maximum retry attempts per object (default: 3)
  * **`:base_delay_ms`** - Base delay for exponential backoff (default: 1000)
  * **`:on_permanent_failure`** - Callback for objects that exceed max retries

### Response
#### Success Response
- **GenServer.on_start()** - The result of starting the GenServer process.
```

--------------------------------

### Start Retry Queue GenServer

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Initiates the retry queue GenServer process. Requires a WeaviateEx client and allows configuration of max retries, base delay for exponential backoff, and a callback for permanent failures.

```Elixir
@spec start_link(keyword()) :: GenServer.on_start()

Start the retry queue GenServer.
## __Options
  * `:client` - WeaviateEx client (required)
  * `:max_retries` - Maximum retry attempts per object (default: 3)
  * `:base_delay_ms` - Base delay for exponential backoff (default: 1000)
  * `:on_permanent_failure` - Callback for objects that exceed max retries
```

--------------------------------

### Get Current Queue Size

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Returns the number of items currently in the retry queue.

```Elixir
@spec size(GenServer.server()) :: non_neg_integer()

Get the current queue size.
```

--------------------------------

### Get Retry Queue State

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Fetches the current internal state of the retry queue GenServer, useful for debugging and testing purposes.

```Elixir
@spec get_state(GenServer.server()) :: map()

Get the current state (for debugging/testing).
```

--------------------------------

### Get Retry Count for UUID

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Retrieves the number of retries attempted for a specific object identified by its UUID.

```Elixir
@spec get_retry_count(GenServer.server(), String.t()) :: non_neg_integer()

Get the retry count for a specific UUID.
```

--------------------------------

### calculate_backoff Function

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Calculates exponential backoff delay in milliseconds for a given retry attempt, capped at 60 seconds. Use this to implement retry strategies with increasing delays.

```elixir
@spec calculate_backoff(non_neg_integer(), pos_integer()) :: pos_integer()
```

--------------------------------

### size

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Returns the current number of items in the retry queue.

```APIDOC
## size(pid)

### Description
Get the current queue size.

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - Required - The server process identifier.

### Response
#### Success Response
- **non_neg_integer()** - The current number of items in the queue.
```

--------------------------------

### calculate_backoff_with_jitter Function

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Calculates exponential backoff with a ±20% jitter. Jitter is useful to prevent simultaneous retries from overwhelming the system.

```elixir
@spec calculate_backoff_with_jitter(non_neg_integer(), pos_integer()) :: pos_integer()
```

--------------------------------

### state() Type Definition

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Defines the state of the retry queue, including the client, queue of failed objects, retry counts, maximum retries, base delay, and a callback for permanent failures.

```elixir
@type state() :: %{
  client: map(),
  queue: :queue.queue(failed_object()),
  retry_counts: %{required(String.t()) => non_neg_integer()},
  max_retries: pos_integer(),
  base_delay_ms: pos_integer(),
  on_permanent_failure: ([failed_object()] -> any()) | nil
}
```

--------------------------------

### clear Function

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Clears all queued objects and resets retry counts for the specified GenServer. Call this to reset the retry queue state.

```elixir
@spec clear(GenServer.server()) :: :ok
```

--------------------------------

### enqueue_failed

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Enqueues a list of failed objects for retry. Objects are tracked by UUID. If an object has already been retried `max_retries` times, the `on_permanent_failure` callback is invoked and the object is dropped.

```APIDOC
## enqueue_failed(pid, failed_objects)

### Description
Enqueues failed objects for retry. If an object has reached `max_retries`, the `on_permanent_failure` callback is invoked.

### Method
GenServer.server()

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - The process ID of the retry queue GenServer.
- **failed_objects** ([failed_object()]) - A list of failed objects to enqueue.

### Response
#### Success Response (ok)
- **:ok** - Indicates the operation was successful.
```

--------------------------------

### enqueue_failed Function

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Enqueues failed objects for retry, tracking them by UUID. If an object exceeds max_retries, the on_permanent_failure callback is invoked.

```elixir
@spec enqueue_failed(GenServer.server(), [failed_object()]) :: :ok
```

--------------------------------

### failed_object() Type Definition

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Defines the structure for a failed object, including its UUID, properties, collection, and optional vector and tenant information.

```elixir
@type failed_object() :: %{
  :uuid => String.t(),
  :properties => map(),
  :collection => String.t(),
  optional(:vector) => [float()],
  optional(:tenant) => String.t()
}
```

--------------------------------

### clear

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Clears all queued objects and resets retry counts for the batch retry queue.

```APIDOC
## clear(pid)

### Description
Clears all queued objects and resets retry counts.

### Method
GenServer.server()

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - The process ID of the retry queue GenServer.

### Response
#### Success Response (ok)
- **:ok** - Indicates the operation was successful.
```

--------------------------------

### rate_limit_error?

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Checks if a given error message indicates a rate limiting issue.

```APIDOC
## rate_limit_error?(message)

### Description
Check if an error message indicates a rate limit error.

### Parameters
#### Path Parameters
- **message** (String.t()) - Required - The error message to check.

### Response
#### Success Response
- **boolean()** - True if the message indicates a rate limit error, false otherwise.
```

--------------------------------

### Check for Rate Limit Error

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Determines if a given error message string indicates a rate limiting issue.

```Elixir
@spec rate_limit_error?(String.t()) :: boolean()

Check if an error message indicates a rate limit error.
```

--------------------------------

### Check if Error is Retryable

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Identifies if an error, represented as a map, can be retried. This includes specific gRPC status codes and rate limit errors.

```Elixir
@spec retryable_error?(map()) :: boolean()

Check if an error is retryable.
Retryable errors include:
  * gRPC status codes: UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED, DEADLINE_EXCEEDED
  * Rate limit errors (detected by message patterns)
```

--------------------------------

### get_state

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Retrieves the current internal state of the retry queue, useful for debugging and testing purposes.

```APIDOC
## get_state(pid)

### Description
Get the current state (for debugging/testing).

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - Required - The server process identifier.

### Response
#### Success Response
- **map()** - The current state of the retry queue.
```

--------------------------------

### drain Function

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Drains all queued objects for manual processing and clears the queue. Returns a list of all pending failed objects.

```elixir
@spec drain(GenServer.server()) :: {:ok, [failed_object()]}
```

--------------------------------

### retryable_error?

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Determines if an error is considered retryable based on predefined criteria, including gRPC status codes and rate limit errors.

```APIDOC
## retryable_error?(error)

### Description
Check if an error is retryable.
Retryable errors include:
  * gRPC status codes: UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED, DEADLINE_EXCEEDED
  * Rate limit errors (detected by message patterns)

### Parameters
#### Path Parameters
- **error** (map()) - Required - The error details to check.

### Response
#### Success Response
- **boolean()** - True if the error is retryable, false otherwise.
```

--------------------------------

### get_retry_count

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Retrieves the number of times an object with a specific UUID has been retried.

```APIDOC
## get_retry_count(pid, uuid)

### Description
Get the retry count for a specific UUID.

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - Required - The server process identifier.
- **uuid** (String.t()) - Required - The unique identifier of the object.

### Response
#### Success Response
- **non_neg_integer()** - The number of retries for the given UUID.
```

--------------------------------

### drain

Source: https://hexdocs.pm/weaviate_ex/WeaviateEx.Batch.RetryQueue.html

Drains all queued objects from the batch retry queue for manual processing.

```APIDOC
## drain(pid)

### Description
Drains all queued objects for manual processing. Returns all pending objects and clears the queue.

### Method
GenServer.server()

### Parameters
#### Path Parameters
- **pid** (GenServer.server()) - The process ID of the retry queue GenServer.

### Response
#### Success Response (ok)
- **{:ok, [failed_object()]}** - A tuple containing :ok and a list of all drained failed objects.
```
Tokens:
Raw