### Start Data Fetcher Jobs in Application Supervisor Source: https://context7.com/qhwa/data_fetcher/llms.txt Configure and start DataFetcher jobs within your application's supervision tree using DataFetcher.child_spec/1. This example shows starting two fetchers with different configurations. ```elixir # lib/my_app/application.ex defmodule MyApp.Application do use Application @impl true def start(_type, _args) do children = [ # Start a data fetcher that refreshes every 20 minutes {DataFetcher, name: :weather_data, fetcher: &MyApp.WeatherService.fetch_current/0, interval: :timer.minutes(20) }, # Start another fetcher with custom cache storage {DataFetcher, name: :exchange_rates, fetcher: {MyApp.ExchangeAPI, :get_rates, [:USD]}, interval: :timer.minutes(5), cache_storage: DataFetcher.CacheStorage.PersistentTerm } ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end # The fetcher module must return {:ok, data} on success defmodule MyApp.WeatherService do def fetch_current do # Fetch from external API case HTTPoison.get("https://api.weather.com/current") do {:ok, %{body: body}} -> {:ok, Jason.decode!(body)} {:error, _} = error -> error end end end ``` -------------------------------- ### DataFetcher Installation Source: https://context7.com/qhwa/data_fetcher/llms.txt Instructions on how to add the DataFetcher library to your Elixir project's dependencies. ```APIDOC ## Installation Add `data_fetcher` to your dependencies in `mix.exs`: ```elixir def deps do [ {:data_fetcher, "~> 0.1.0"} ] end ``` ``` -------------------------------- ### Integrate DataFetcher in Application Source: https://context7.com/qhwa/data_fetcher/llms.txt Full example demonstrating supervision tree integration, fetcher logic with error handling, and result retrieval in a controller. ```elixir # lib/my_app/application.ex defmodule MyApp.Application do use Application @impl true def start(_type, _args) do children = [ # External API data with retry on failure {DataFetcher, name: :github_stats, fetcher: &MyApp.GitHub.fetch_repo_stats/0, interval: :timer.minutes(30) }, # Database aggregate that rarely changes {DataFetcher, name: :user_metrics, fetcher: &MyApp.Metrics.aggregate_users/0, interval: :timer.hours(1), cache_storage: DataFetcher.CacheStorage.PersistentTerm }, # Fast-updating configuration {DataFetcher, name: :feature_flags, fetcher: {MyApp.Config, :load_flags, [:production]}, interval: :timer.minutes(1) } ] Supervisor.start_link(children, strategy: :one_for_one) end end # lib/my_app/github.ex defmodule MyApp.GitHub do def fetch_repo_stats do url = "https://api.github.com/repos/owner/repo" headers = [{"Authorization", "token #{github_token()}"}] case HTTPoison.get(url, headers) do {:ok, %{status_code: 200, body: body}} -> {:ok, Jason.decode!(body)} {:ok, %{status_code: code}} -> {:error, "GitHub API returned #{code}"} {:error, %{reason: reason}} -> {:error, reason} end end defp github_token, do: Application.get_env(:my_app, :github_token) end # lib/my_app_web/controllers/stats_controller.ex defmodule MyAppWeb.StatsController do use MyAppWeb, :controller def index(conn, _params) do json(conn, %{ github: DataFetcher.result(:github_stats), users: DataFetcher.result(:user_metrics), features: DataFetcher.result(:feature_flags) }) end end ``` -------------------------------- ### DataFetcher.result/1 Usage Source: https://context7.com/qhwa/data_fetcher/llms.txt Explains how to retrieve cached data using `DataFetcher.result/1` and provides examples for use in controllers and LiveViews. ```APIDOC ## DataFetcher.result/1 Retrieves the cached result from a named fetcher. If called during the initial fetch (before any data is available), it will block and wait until the first successful fetch completes. After the initial fetch, results are read directly from the cache and returned immediately. ```elixir # Basic usage - get cached weather data weather = DataFetcher.result(:weather_data) # => %{"temperature" => 72, "conditions" => "sunny", "humidity" => 45} # Use in a Phoenix controller defmodule MyAppWeb.WeatherController do use MyAppWeb, :controller def index(conn, _params) do weather = DataFetcher.result(:weather_data) json(conn, %{ current: weather, cached_at: DateTime.utc_now() }) end end # Use in a LiveView defmodule MyAppWeb.DashboardLive do use MyAppWeb, :live_view def mount(_params, _session, socket) do rates = DataFetcher.result(:exchange_rates) {:ok, assign(socket, rates: rates)} end end ``` ``` -------------------------------- ### Get Fetched Result Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Retrieve the latest fetched data using `DataFetcher.result/1`, passing the name assigned to the fetcher in the supervisor spec. ```elixir DataFetcher.result(:my_fetcher) ``` -------------------------------- ### Implement Custom Cache Storage Source: https://context7.com/qhwa/data_fetcher/llms.txt Create a custom storage adapter by implementing the DataFetcher.CacheStorage behaviour with init/1, put/2, and get/1 callbacks. ```elixir defmodule MyApp.RedisCacheStorage do @behaviour DataFetcher.CacheStorage @impl true def init(opts) do # Initialize connection or resources name = opts[:name] # Setup logic here :ok end @impl true def put(name, result) do key = "data_fetcher:#{name}" {:ok, _} = Redix.command(:redix, ["SET", key, :erlang.term_to_binary(result)]) :ok end @impl true def get(name) do key = "data_fetcher:#{name}" case Redix.command(:redix, ["GET", key]) do {:ok, nil} -> nil {:ok, binary} -> {:ok, :erlang.binary_to_term(binary)} {:error, _} -> nil end end end # Usage {DataFetcher, name: :distributed_cache, fetcher: &MyApp.DataSource.fetch/0, interval: :timer.minutes(15), cache_storage: MyApp.RedisCacheStorage } ``` -------------------------------- ### Configure DataFetcher Instances Source: https://context7.com/qhwa/data_fetcher/llms.txt Define per-fetcher configurations for specific data sources and set global default intervals. ```elixir {DataFetcher, name: :large_dataset, fetcher: &MyApp.BigData.fetch/0, interval: :timer.hours(24), cache_storage: DataFetcher.CacheStorage.PersistentTerm # Zero-copy reads for huge data } {DataFetcher, name: :frequently_updated, fetcher: &MyApp.LiveStats.fetch/0, interval: :timer.seconds(10), cache_storage: DataFetcher.CacheStorage.Ets # Better for frequent writes } # Set default interval globally config :data_fetcher, :default_interval, :timer.minutes(10) ``` -------------------------------- ### Define Fetcher Options: Module Callback Source: https://context7.com/qhwa/data_fetcher/llms.txt Configure a DataFetcher job by providing a module that implements the fetch/0 callback. The module's fetch/0 function must return {:ok, data}. ```elixir # Option 3: Module with fetch/0 callback defmodule MyApp.ProductCatalog do def fetch do products = MyApp.Repo.all(MyApp.Product) {:ok, Enum.map(products, &format_product/1)} end defp format_product(product) do %{id: product.id, name: product.name, price: product.price} end end {DataFetcher, name: :products, fetcher: MyApp.ProductCatalog, interval: :timer.hours(1) } ``` -------------------------------- ### Configure Cache Storage Adapter (ETS) Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Configure DataFetcher to use ETS as the cache storage. This is the default and generally recommended for most use cases. ```elixir # file: config/config.exs # with ETS (by default) config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.Ets ``` -------------------------------- ### Configure Cache Storage Adapter (Persistent Term) Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Configure DataFetcher to use `persistent_term` for cache storage. This can offer better read performance by avoiding message copying but is slower for writes. ```elixir # file: config/config.exs # or with persistent_term config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.PersistentTerm ``` -------------------------------- ### Define Fetcher Options: MFA Tuple Source: https://context7.com/qhwa/data_fetcher/llms.txt Configure a DataFetcher job using an MFA (Module, Function, Arguments) tuple. This allows specifying a function with arguments to be called for fetching data. ```elixir # Option 2: MFA tuple {Module, Function, Arguments} {DataFetcher, name: :api_fetch, fetcher: {MyApp.APIClient, :fetch_data, [:users, %{limit: 100}]}, interval: :timer.minutes(10) } ``` -------------------------------- ### Cache Storage Configuration Source: https://context7.com/qhwa/data_fetcher/llms.txt Explains the configuration options for cache storage adapters, ETS (default) and persistent_term. ```APIDOC ## Cache Storage Configuration DataFetcher supports two cache storage adapters: ETS (default) for general use and persistent_term for large, rarely-changing data. ETS provides good read/write performance while persistent_term offers zero-copy reads at the cost of slower writes with global GC. ```elixir # Global configuration in config/config.exs # Use ETS (default) config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.Ets # Or use persistent_term for large, rarely-changing data config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.PersistentTerm ``` ``` -------------------------------- ### Define a Fetching Worker Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Create a worker module with a `fetch/1` function that performs the data fetching. It should return `{:ok, data}` on success. ```elixir defmodule MyWorker do def fetch do # maybe do some networking # if successful: {:ok, %{foo: "bar"}} end end ``` -------------------------------- ### DataFetcher.child_spec/1 Usage Source: https://context7.com/qhwa/data_fetcher/llms.txt Demonstrates how to use `DataFetcher.child_spec/1` to add a data fetcher job to your application's supervision tree, including configuration options. ```APIDOC ## DataFetcher.child_spec/1 Builds the supervisor child spec to start a data fetcher job. This is the primary way to add DataFetcher to your application's supervision tree. It accepts configuration options for naming the fetcher, defining the data source, setting the refresh interval, and selecting the cache storage adapter. ```elixir # lib/my_app/application.ex defmodule MyApp.Application do use Application @impl true def start(_type, _args) do children = [ # Start a data fetcher that refreshes every 20 minutes {DataFetcher, name: :weather_data, fetcher: &MyApp.WeatherService.fetch_current/0, interval: :timer.minutes(20) }, # Start another fetcher with custom cache storage {DataFetcher, name: :exchange_rates, fetcher: {MyApp.ExchangeAPI, :get_rates, [:USD]}, interval: :timer.minutes(5), cache_storage: DataFetcher.CacheStorage.PersistentTerm } ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end # The fetcher module must return {:ok, data} on success defmodule MyApp.WeatherService do def fetch_current do # Fetch from external API case HTTPoison.get("https://api.weather.com/current") do {:ok, %{body: body}} -> {:ok, Jason.decode!(body)} {:error, _} = error -> error end end end ``` ``` -------------------------------- ### Define Fetcher Options: Anonymous Function Source: https://context7.com/qhwa/data_fetcher/llms.txt Configure a DataFetcher job using an anonymous function as the fetcher. The function must return {:ok, data} and will be called at the specified interval. ```elixir # Option 1: Anonymous function (arity 0) {DataFetcher, name: :simple_fetch, fetcher: fn -> {:ok, System.system_time()} end, interval: :timer.seconds(30) } ``` -------------------------------- ### Add DataFetcher to Mix Dependencies Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Include the data_fetcher package in your project's dependencies in mix.exs. ```elixir def deps do [ {:data_fetcher, "~> 0.1.0"} ] end ``` -------------------------------- ### Fetcher Definition Options Source: https://context7.com/qhwa/data_fetcher/llms.txt Details the three supported formats for defining fetcher functions: anonymous functions, MFA tuples, and modules. ```APIDOC ## Fetcher Definition Options The fetcher option supports three formats: anonymous functions, MFA tuples, and modules. Each format must return `{:ok, data}` on success. Failed fetches are automatically retried on the next interval. ```elixir # Option 1: Anonymous function (arity 0) {DataFetcher, name: :simple_fetch, fetcher: fn -> {:ok, System.system_time()} end, interval: :timer.seconds(30) } # Option 2: MFA tuple {Module, Function, Arguments} {DataFetcher, name: :api_fetch, fetcher: {MyApp.APIClient, :fetch_data, [:users, %{limit: 100}]}, interval: :timer.minutes(10) } # Option 3: Module with fetch/0 callback defmodule MyApp.ProductCatalog do def fetch do products = MyApp.Repo.all(MyApp.Product) {:ok, Enum.map(products, &format_product/1)} end defp format_product(product) do %{id: product.id, name: product.name, price: product.price} end end {DataFetcher, name: :products, fetcher: MyApp.ProductCatalog, interval: :timer.hours(1) } ``` ``` -------------------------------- ### Configure Cache Storage: PersistentTerm Source: https://context7.com/qhwa/data_fetcher/llms.txt Globally configure DataFetcher to use persistent_term for cache storage. This is recommended for large, rarely-changing data due to zero-copy reads. ```elixir # Or use persistent_term for large, rarely-changing data config :data_fetcher, :cache_storage, DataFetcher.CacheStorage.PersistentTerm ``` -------------------------------- ### Add DataFetcher to Supervisor Tree Source: https://github.com/qhwa/data_fetcher/blob/master/README.md Integrate DataFetcher into your Elixir application's supervisor tree by providing a specification. The `fetcher` should be a module and function that returns data, and `interval` defines the fetching frequency. ```elixir defmodule MyApp.Application do use Application @impl true def start(_type, _args) do children = [ {DataFetcher, my_fetcher_spec()}, ... ] opts = [strategy: :one_for_one, name: MySupervisor] Supervisor.start_link(children, opts) end defp my_fetcher_spec, do: [ name: :my_fetcher, fetcher: &MyWorker.fetch/1, interval: :timer.minutes(20) ] end ``` -------------------------------- ### Retrieve Cached Data with DataFetcher.result/1 Source: https://context7.com/qhwa/data_fetcher/llms.txt Access cached data from a named fetcher using DataFetcher.result/1. This function blocks until the first fetch is complete if no data is yet available. ```elixir # Basic usage - get cached weather data weather = DataFetcher.result(:weather_data) # => %{"temperature" => 72, "conditions" => "sunny", "humidity" => 45} # Use in a Phoenix controller defmodule MyAppWeb.WeatherController do use MyAppWeb, :controller def index(conn, _params) do weather = DataFetcher.result(:weather_data) json(conn, %{ current: weather, cached_at: DateTime.utc_now() }) end end # Use in a LiveView defmodule MyAppWeb.DashboardLive do use MyAppWeb, :live_view def mount(_params, _session, socket) do rates = DataFetcher.result(:exchange_rates) {:ok, assign(socket, rates: rates)} end end ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.