DataFetcher

https://hexdocs.pm/data_fetcher
Admin
DataFetcher is an Erlang library that provides an abstraction for periodic data fetching jobs with...
Tokens:4,573
Snippets:51
Trust Score:9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Start DataFetcher with custom options

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-child_spec.html

Example of starting a DataFetcher as a child in a supervisor. This snippet shows how to set a unique name, a custom fetcher function, and a specific interval.

```elixir
defp my_fetcher_spec,
    do: {
        DataFetcher,
        name: :my_fetcher,
        fetcher: &MyWorker.fetch/1,
        interval: :timer.minutes(20)
      }
```

--------------------------------

### DataFetcher Example Usage in Application

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-child_spec.html

An example of how to integrate the DataFetcher into your application's supervision tree.

```APIDOC
## Example Usage

### Description
This example shows how to include a DataFetcher child spec in your `MyApp.Application`'s children list.

### Code
```elixir
# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      my_fetcher_spec(),
      ...
    ]

    opts = [strategy: :one_for_one, name: MySupervisor]
    Supervisor.start_link(children, opts)
  end

  defp my_fetcher_spec,
    do: {
        DataFetcher,
        name: :my_fetcher,
        fetcher: &MyWorker.fetch/1,
        interval: :timer.minutes(20)
      }
end
```
```

--------------------------------

### DataFetcher.child_spec/1

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Builds the supervisor child specification to start a data fetcher process.

```APIDOC
## child_spec(opts)

### Description
Builds the supervisor child spec to start the data_fetcher.

### Parameters
#### Request Body
- **name** (atom) - Required - The identity of the job, should be unique.
- **fetcher** (fetcher_def) - Required - Definition of how data is fetched from the source. Must return {:ok, data}.
- **interval** (non_neg_integer) - Optional - Wait time in ms between fetches. Default is 10 minutes.
- **cache_storage** (atom) - Optional - Cache storage adapter (e.g., ETS or persistent_term). Default is ETS.
```

--------------------------------

### Implement DataFetcher in Application

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Example of integrating DataFetcher into an Elixir application supervisor tree.

```elixir
# lib/my_app/application.ex
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      my_fetcher_spec(),
      ...
    ]

    opts = [strategy: :one_for_one, name: MySupervisor]
    Supervisor.start_link(children, opts)
  end

  defp my_fetcher_spec,
    do: {
        DataFetcher,
        name: :my_fetcher,
        fetcher: &MyWorker.fetch/1,
        interval: :timer.minutes(20)
      }
end
```

--------------------------------

### Retrieve fetcher result

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-result.html

Demonstrates starting a DataFetcher supervisor and retrieving the result by name. The function waits for completion if the fetch is in progress.

```elixir
iex> opts = [
...>   name: :function_fetcher,
...>   fetcher: fn -> {:ok, %{foo: 1}} end
...> ]
...>
...> {:ok, _} = Supervisor.start_link([{DataFetcher, opts}], strategy: :one_for_one)
...>
...> DataFetcher.result(:function_fetcher)
%{foo: 1}
```

--------------------------------

### DataFetcher - child_spec/1

Source: https://hexdocs.pm/data_fetcher/index.html

Builds the supervisor child specification for starting a DataFetcher process. It supports various options for configuring the fetcher's behavior, including name, fetcher definition, interval, and cache storage.

```APIDOC
## POST /api/users

### Description
Build the supervisor child spec to start the data_fetcher.

### Method
POST

### Endpoint
/api/users

### Parameters
#### Path Parameters
- **userId** (integer) - Required - The ID of the user to retrieve.

#### Query Parameters
- **include_details** (boolean) - Optional - Whether to include detailed user information.

#### Request Body
- **username** (string) - Required - The username for the new user.
- **email** (string) - Required - The email address for the new user.

### Request Example
```json
{
  "username": "johndoe",
  "email": "john.doe@example.com"
}
```

### Response
#### Success Response (200)
- **id** (integer) - The unique identifier for the created user.
- **username** (string) - The username of the created user.
- **email** (string) - The email address of the created user.

#### Response Example
```json
{
  "id": 123,
  "username": "johndoe",
  "email": "john.doe@example.com"
}
```
```

--------------------------------

### Get data from cache

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage-callback-get.html

Retrieves data from the cache by its name. Returns the cached result or nil if not found.

```Elixir
get(name :: atom()) :: {:ok, result :: any()} | nil
```

--------------------------------

### DataFetcher Result Example

Source: https://hexdocs.pm/data_fetcher/index.html

Retrieves the result of a data fetch job. If the data is still being fetched on the first round, it waits for successful completion. Otherwise, it returns cached data immediately.

```elixir
iex> opts = [
...>   name: :function_fetcher,
...>   fetcher: fn -> {:ok, %{foo: 1}} end
...> ]
...>
...> {:ok, _} = Supervisor.start_link([{DataFetcher, opts}], strategy: :one_for_one)
...
...> DataFetcher.result(:function_fetcher)
%{foo: 1}
```

--------------------------------

### DataFetcher child_spec Function

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-child_spec.html

This section describes the `child_spec` function of the DataFetcher module, which is used to build a supervisor child specification for starting the data fetcher.

```APIDOC
## DataFetcher.child_spec/1

### Description
Build the supervisor child spec to start the data_fetcher.

### Method
N/A (Function definition)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Supported Options
- **`:name`** (atom) - Required - The identity of the job, should be unique.
- **`:fetcher`** (module function) - Required - Definition of how data is fetched from the source. A fetcher must return `{:ok, data}` if successful.
- **`:interval`** (integer) - Optional - A value in `ms` indicating how long it waits until next fetching. The execution time will not affect the interval, neither does the failure of a fetching. 10 minutes by default.
- **`:cache_storage`** (module) - Optional - What cache storage adapter to store the data. Ets by default. See Cache Storage section below.

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### DataFetcher.CacheStorage.init

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage-callback-init.html

Initializes the cache storage with a given keyword.

```APIDOC
## init(keyword)

### Description
Initializes the cache storage.

### Method
N/A (Callback function)

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
`:ok` - Indicates successful initialization.

#### Error Response
`{:error, any()}` - Indicates an error during initialization, with details provided in the tuple.

#### Response Example
```
:ok
```

```
{:error, "Initialization failed"}
```
```

--------------------------------

### Initialize cache storage

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage.html

Initializes the cache storage with a given keyword. Returns :ok on success or an error tuple.

```elixir
init(keyword()) :: :ok | {:error, any()}
```

--------------------------------

### DataFetcher.CacheStorage.PersistentTerm

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage.PersistentTerm.html

Overview of the PersistentTerm storage adapter implementation.

```APIDOC
## PersistentTerm Storage Adapter

### Description
The `DataFetcher.CacheStorage.PersistentTerm` module serves as a storage adapter for caching mechanisms within the `data_fetcher` library (v0.1.0). It utilizes Erlang's persistent_term for efficient, read-heavy data storage.
```

--------------------------------

### DataFetcher.CacheStorage Module

Source: https://hexdocs.pm/data_fetcher/api-reference.html

Defines the cache storage behavior.

```APIDOC
## DataFetcher.CacheStorage Module

### Description
Cache storage behaviour.

### Sub-modules
- DataFetcher.CacheStorage.Ets
- DataFetcher.CacheStorage.PersistentTerm
```

--------------------------------

### DataFetcher.CacheStorage Callbacks

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage.html

This section outlines the available callbacks for interacting with the CacheStorage.

```APIDOC
## GET /cache_storage/get

### Description
Retrieves a cached result by its name.

### Method
GET

### Endpoint
/cache_storage/get

### Parameters
#### Query Parameters
- **name** (atom) - Required - The name of the cached item to retrieve.

### Response
#### Success Response (200)
- **result** (any) - The cached result if found.

#### Response Example
```json
{
  "result": "some_value"
}
```

## INIT /cache_storage/init

### Description
Initializes the cache storage with a given keyword.

### Method
POST

### Endpoint
/cache_storage/init

### Parameters
#### Request Body
- **keyword** (keyword()) - Required - The keyword to use for initialization.

### Response
#### Success Response (200)
- **status** (atom) - Indicates success (:ok) or failure (:error).

#### Response Example
```json
{
  "status": ":ok"
}
```

## PUT /cache_storage/put

### Description
Stores a result in the cache storage with a given name.

### Method
PUT

### Endpoint
/cache_storage/put

### Parameters
#### Request Body
- **name** (atom) - Required - The name to associate with the cached result.
- **result** (any) - Required - The result to cache.

### Response
#### Success Response (200)
- **status** (atom) - Indicates success (:ok) or failure (:error).

#### Response Example
```json
{
  "status": ":ok"
}
```
```

--------------------------------

### Define option type

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Defines the configuration options available for the fetcher.

```elixir
option() ::
  {:name, atom()}
  | {:cache_storage, atom()}
  | {:fetcher, fetcher_def()}
  | {:interval, ms :: non_neg_integer()}
```

--------------------------------

### DataFetcher Module

Source: https://hexdocs.pm/data_fetcher/api-reference.html

Provides an abstraction on periodic data fetching jobs.

```APIDOC
## DataFetcher Module

### Description
An abstraction on periodic data fetching jobs.

### Sub-modules
- DataFetcher.CacheStorage
```

--------------------------------

### DataFetcher.CacheStorage.Ets Adapter

Source: https://hexdocs.pm/data_fetcher/api-reference.html

Implements ETS storage adapter for cache.

```APIDOC
## DataFetcher.CacheStorage.Ets Adapter

### Description
Ets storage adapter
```

--------------------------------

### DataFetcher Cache Storage Configuration

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-child_spec.html

Details on configuring the cache storage for the DataFetcher, including default ETS and alternative PersistentTerm adapter.

```APIDOC
## DataFetcher Cache Storage

### Description
When data is fetched from the source, it will be saved in the cache storage. By default it uses an ETS table so that later reads can directly read from the ETS table. This is adequate for most cases unless the data size is huge. In such large data cases, copying and messaging between processes can be expensive and have performance penalties.

If the data rarely or never changes, you can improve the performance in such cases by employing the `persistent_term` adapter, which is faster at reading no matter how large the data is. Be aware that on the other hand, writing into the cache storage is slower for `persistent_term` because global GC will happen for every write.

### Configuration

#### Global Configuration
```elixir
config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.PersistentTerm
```

#### Per-Fetcher Configuration
Use the `:cache_storage` option when defining the fetcher:
```elixir
{DataFetcher, [..., cache_storage: DataFetcher.CacheStorage.PersistentTerm, ...]}`
```
```

--------------------------------

### DataFetcher.CacheStorage.PersistentTerm Adapter

Source: https://hexdocs.pm/data_fetcher/api-reference.html

Implements persistent term storage adapter for cache.

```APIDOC
## DataFetcher.CacheStorage.PersistentTerm Adapter

### Description
persistent_term storage adapter
```

--------------------------------

### Configure DataFetcher cache storage globally

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-child_spec.html

Change the cache storage adapter globally for DataFetcher. This is useful for large datasets where persistent_term offers better read performance.

```elixir
config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.PersistentTerm
```

--------------------------------

### Put data into cache

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage.html

Stores a result in the cache associated with a given name. Returns :ok on success or an error tuple.

```elixir
put(name :: atom(), result :: any()) :: :ok | {:error, any()}
```

--------------------------------

### DataFetcher.CacheStorage.get

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage-callback-get.html

Retrieves cached data by its name. Returns the cached result if found, otherwise returns nil.

```APIDOC
## DataFetcher.CacheStorage.get

### Description
Retrieves cached data by its name. Returns the cached result if found, otherwise returns nil.

### Method
GET (implied, as it's a retrieval operation)

### Endpoint
/websites/hexdocs_pm_data_fetcher/cache_storage/get

### Parameters
#### Path Parameters
- **name** (atom) - Required - The name of the cache entry to retrieve.

### Response
#### Success Response (200)
- **result** (any) - The cached data if found.

#### Response Example
```json
{:ok, "cached_value"}
```

#### Nil Response
- **nil** - Returned when the cache entry is not found.

#### Nil Response Example
```
nil
```
```

--------------------------------

### put(name, result)

Source: https://hexdocs.pm/data_fetcher/DataFetcher.CacheStorage-callback-put.html

Stores a result in the cache associated with the given name.

```APIDOC
## put(name, result)

### Description
Stores a result in the cache associated with the given name.

### Parameters
- **name** (atom) - Required - The identifier for the cache entry.
- **result** (any) - Required - The data to be stored.

### Response
- **Success** (:ok) - Returned when the operation is successful.
- **Error** ({:error, any()}) - Returned if the operation fails.
```

--------------------------------

### Define result function

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Defines the specification for retrieving the result of a fetch job.

```elixir
result(fetcher_name :: atom()) :: any()
```

--------------------------------

### Define child_spec function

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Defines the supervisor child specification for the fetcher.

```elixir
child_spec([option()]) :: Supervisor.child_spec()
```

--------------------------------

### DataFetcher.result/1

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Retrieves the latest result of a specific fetcher job.

```APIDOC
## result(name)

### Description
Get result of the fetch. If the data is still being fetched at the first round, it will wait until the data is successfully fetched before returning the result.

### Parameters
#### Path Parameters
- **name** (atom) - Required - The identifier of the fetch job.

### Response
#### Success Response (200)
- **result** (any) - The fetched data stored in the cache.
```

--------------------------------

### Define fetcher_def type

Source: https://hexdocs.pm/data_fetcher/DataFetcher.html

Defines the specification for a fetcher function or module.

```elixir
fetcher_def() :: (() -> {:ok, any()}) | mfa() | module()
```

--------------------------------

### DataFetcher.result API

Source: https://hexdocs.pm/data_fetcher/DataFetcher-function-result.html

Retrieves the result of a data fetch operation. If the data is still being fetched, it waits for completion. Otherwise, it returns cached data.

```APIDOC
## DataFetcher.result

### Description
Retrieves the result of a data fetch operation. If the data is still being fetched, it waits for completion. Otherwise, it returns cached data.

### Method
GET (conceptual, as it's a function call)

### Endpoint
N/A (Function call within a module)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **name** (atom) - Required - The identifier of the fetch job.

### Returns
- **any** - The result of the data fetch operation.

### Response Example
```json
{
  "example": "%{foo: 1}"
}
```

### Example
```elixir
opts = [
  name: :function_fetcher,
  fetcher: fn -> {:ok, %{foo: 1}} end
]

{:ok, _} = Supervisor.start_link([{DataFetcher, opts}], strategy: :one_for_one)

DataFetcher.result(:function_fetcher)
# => %{foo: 1}
```
```

--------------------------------

### DataFetcher - result/1

Source: https://hexdocs.pm/data_fetcher/index.html

Retrieves the result of a data fetch job. If the data is currently being fetched, this function will wait for the fetch to complete. Otherwise, it returns the cached data immediately.

```APIDOC
## GET /api/users/{userId}

### Description
Get result of the fetch.

### Method
GET

### Endpoint
/api/users/{userId}

### Parameters
#### Path Parameters
- **name** (atom) - Required - The identifier of the fetch job.

### Response
#### Success Response (200)
- **any** - The result of the fetch operation.

### Response Example
```json
{
  "data": "some_fetched_data"
}
```
```
Tokens:
Raw