### Configure Application Start Phases

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Seeding Metrics.md

Configure your project's `mix.exs` to include a custom start phase for seeding PromEx telemetry.

```elixir
def application do
  [
    extra_applications: [],
    mod: {MyApp.Application, []},
    start_phases: [seed_prom_ex_telemetry: []]
  ]
end
```

--------------------------------

### Fetch Metrics Endpoint

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

After setup, you can access your application's metrics by curling the `/metrics` endpoint. This example shows sample output.

```bash
$ curl localhost:4000/metrics
# HELP my_cool_app_application_dependency_info Information regarding the application's dependencies.
# TYPE my_cool_app_application_dependency_info gauge
my_cool_app_application_dependency_info{modules="69",name="hex",version="0.20.5"} 1
my_cool_app_application_dependency_info{modules="1",name="connection",version="1.0.4"} 1
my_cool_app_application_dependency_info{modules="4",name="telemetry_poller",version="0.5.1"} 1
...
```

--------------------------------

### Run ab_graph_scale.sh Example

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Example command to execute ab_graph_scale.sh for running multiple tests with specified concurrent connection ranges, request counts, keepalive, and wait times.

```bash
./ab-graph_scale.sh -c 10 -e 50 -k -s 10 -n 1000 -u https://www.test.com -w 30
```

--------------------------------

### Run ab_graph.sh Example

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Example command to execute the ab_graph.sh script for testing a URL with specified requests, concurrent connections, and keepalive enabled.

```bash
./ab-graph.sh -u http://www.testsite.com/ -n 500 -c 100 -k
```

--------------------------------

### Implement Start Phase Callback

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Seeding Metrics.md

Add a `start_phase` callback to your application's main module to trigger the seeding of event metrics.

```elixir
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      MyApp.PromEx
      # ... Repo, Phoenix, etc
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  @impl true
  def start_phase(:seed_prom_ex_telemetry, :normal, _) do
    MyApp.PromEx.seed_event_metrics()
    :ok
  end
end
```

--------------------------------

### ab_graph_merge.sh Example

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Example command to merge results from two separate ab_graph.sh test runs into a single graph for comparison.

```bash
./ab-graph_merge.sh $PWD/results/www.testsite.com/2020-04-23-00-30-12 $PWD/results/www.testsite.com/2020-04-22-23-34-50
```

--------------------------------

### ApacheBench Test Summary Output

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Example output from an ApacheBench test, showing server information, test parameters, and performance metrics like requests per second and connection times.

```text
This is ApacheBench, Version 2.3 <$Revision: 1843412 $
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.testsite.com (be patient)


Server Software:        nginx/1.10.3
Server Hostname:        www.testsite.com
Server Port:            80

Document Path:          //
Document Length:        4 bytes

Concurrency Level:      100
Time taken for tests:   26.012 seconds
Complete requests:      500
Failed requests:        0
Non-2xx responses:      500
Keep-Alive requests:    0
Total transferred:      200000 bytes
HTML transferred:       2000 bytes
Requests per second:    19.22 [#/sec] (mean)
Time per request:       5202.324 [ms] (mean)
Time per request:       52.023 [ms] (mean, across all concurrent requests)
Transfer rate:          7.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.3      0       7
Processing:   185 4926 1699.9   4718   10431
Waiting:      180 4925 1699.8   4718   10431
Total:        185 4926 1700.2   4718   10434

Percentage of the requests served within a certain time (ms)
  50%   4718
  66%   5203
  75%   5587
  80%   5781
  90%   7518
  95%   8573
  98%   9825
  99%  10228
 100%  10434 (longest request)

```

--------------------------------

### Add PromEx to Application Supervisor

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Include the generated PromEx module in your `application.ex` supervision tree. It should be started before other processes to capture init events.

```elixir
defmodule MyCoolApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      # PromEx should be started before anything else as PromEx will caputre init events from
      # libraries like Ecto, Phoenix and Oban. If it is started after those other supervision trees
      # you will miss those events and metrics.
      MyCoolApp.PromEx,

      MyCoolApp.Repo,
      MyCoolApp.Endpoint
      ...
    ]

    opts = [strategy: :one_for_one, name: MyCoolApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

--------------------------------

### Implement Event Metrics in a PromEx Plugin

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Writing PromEx Plugins.md

Implement the `event_metrics/1` callback to expose event-based metrics. This example shows how to capture HTTP request duration using `distribution` and `Telemetry.Metrics`.

```elixir
defmodule MyApp.PromEx.Plugins.MyPhoenix do
  use PromEx.Plugin

  @impl true
  def event_metrics(opts) do
    http_metrics_tags = gen_http_metrics_tags(opts)
    phoenix_router = get_phoenix_router(opts)
    phoenix_stop_event = [:phoenix, :endpoint, :stop]

    Event.build(
      :phoenix_http_event_metrics,
      [
        # Capture request duration information
        distribution(
          [:phoenix, :http, :request, :duration, :milliseconds],
          event_name: phoenix_stop_event,
          measurement: :duration,
          description: "The time it takes for the application to respond to HTTP requests.",
          reporter_options: [
            buckets: exponential!(1, 2, 12)
          ],
          tag_values: get_conn_tags(phoenix_router),
          tags: http_metrics_tags,
          unit: {:native, :millisecond}
        )

        # Additional event based metrics ...
      ]
    )
  end
end
```

--------------------------------

### Define Manual Metrics in a PromEx Plugin

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Writing PromEx Plugins.md

This snippet shows how to define manual metrics within a PromEx plugin. The `manual_metrics/1` function is called on application start, and metrics are updated via `PromEx.ManualMetricsManager.refresh_metrics/1`.

```elixir
defmodule PromEx.Plugins.Application do
  use PromEx.Plugin

  @impl true
  def manual_metrics(opts) do
    otp_app = Keyword.fetch!(opts, :otp_app)
    apps = Keyword.get(opts, :deps, :all)

    Manual.build(
      :application_versions_manual_metrics,
      {__MODULE__, :apps_running, [otp_app, apps]},
      [
        # Capture information regarding the primary application (i.e the user's application)
        last_value(
          [otp_app | [:application, :primary, :info]],
          event_name: [otp_app | [:application, :primary, :info]],
          description: "Information regarding the primary application.",
          measurement: :status,
          tags: [:name, :version, :modules]
        )

        # Additional metrics here
      ]
    )
  end

  @doc false
  def apps_running(otp_app, apps) do
    ...

    # Emit primary app details
    :telemetry.execute(
      [otp_app | [:application, :primary, :info]],
      %{ 
        status: if(Map.has_key?(started_apps, otp_app), do: 1, else: 0)
      },
      %{ 
        name: otp_app,
        version:
          Map.get_lazy(started_apps, otp_app, fn ->
            Map.get(loaded_only_apps, otp_app, "undefined")
          end),
        modules: length(Application.spec(otp_app)[:modules])
      }
    )
  end
end
```

--------------------------------

### ab_graph_scale.sh Usage Options

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Displays the available command-line options for the ab_graph_scale.sh script, used for running multiple ab_graph.sh tests with varying concurrent connections.

```bash
Usage: ./ab-graph_scale.sh OPTIONS

OPTIONS:
-c    Start concurrent connections at        (default: 5)
-e    Stop concurrent connections at         (default: 25)
-k    Enable keepalive connections           (default: no)
-s    Concurrent connections increment step  (default: 5)
-n    Number of requests                     (default: 500)
-u    Url to test                            (mandatory)
-w    Wait time between tests in seconds.    (default: 60s)
```

--------------------------------

### ab_graph.sh Usage Options

Source: https://github.com/akoutmos/prom_ex/blob/master/benchmarks/README.md

Displays the available command-line options for the ab_graph.sh script, which runs ApacheBench tests and generates plots.

```bash
Usage: $0 OPTIONS

OPTIONS:
-c    Concurrent connections  (default: 1)
-k    Enable keepalive        (default: no)
-E    Extra parameters
-n    Number of requests      (default: 1)
-u    Url to test             (mandatory)
-h    Print help.
-V    Debug mode.
```

--------------------------------

### Connect to Running Container

Source: https://github.com/akoutmos/prom_ex/blob/master/example_applications/web_app/README.md

Use these commands to access the shell of a running WebApp container and then connect to the Elixir application console.

```bash
$ docker exec -it <CONTAINER ID> bash
$ iex --sname console --cookie super_secret --remsh elixir_app_{one|two}
```

--------------------------------

### ApacheBench Stress Test with PromEx

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Results from an ApacheBench stress test showing performance metrics when PromEx is active. This serves as a baseline for performance comparison.

```terminal
$ ./benchmarks/ab-graph.sh -u http://localhost:4000 -n 1000 -c 50 -k
Server Software:        Cowboy
Server Hostname:        localhost
Server Port:            4000

Document Path:          /
Document Length:        3389 bytes

Concurrency Level:      50
Time taken for tests:   4.144 seconds
Complete requests:      1000
Failed requests:        0
Keep-Alive requests:    1000
Total transferred:      4060000 bytes
HTML transferred:       3389000 bytes
Requests per second:    241.32 [#/sec] (mean)
Time per request:       207.191 [ms] (mean)
Time per request:       4.144 [ms] (mean, across all concurrent requests)
Transfer rate:          956.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:    39  202  24.3    203     264
Waiting:       38  202  24.3    203     264
Total:         39  202  24.2    203     264

Percentage of the requests served within a certain time (ms)
  50%    203
  66%    210
  75%    215
  80%    218
  90%    227
  95%    237
  98%    246
  99%    255
 100%    264 (longest request)

```

--------------------------------

### Generate PromEx Configuration

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Run this mix task to generate the PromEx configuration file. Replace `YOUR_PROMETHEUS_DATASOURCE_ID` with your Grafana Prometheus data source name.

```bash
$ mix prom_ex.gen.config --datasource YOUR_PROMETHEUS_DATASOURCE_ID
```

--------------------------------

### Generate Distinct PromEx Configurations

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Running Multiple Agents.md

Use the mix prom_ex.gen.config task to create separate configurations for different agents. Specify a unique directory and module name for each.

```sh
mix prom_ex.gen.config -d ops -m PromExOps
mix prom_ex.gen.config -d biz -m PromExBiz
```

--------------------------------

### Configure Multiple PromEx Agents

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Running Multiple Agents.md

Configure each PromEx module with unique `working_directory`, `agent_port`, and `grpc_port` to prevent collisions when running multiple agents.

```elixir
config :my_app, MyApp.PromExOps,
  grafana_agent: [
    working_directory: System.fetch_env!("RELEASE_TMP") <> "/grafana-ops",
    config_opts: [
      ...
      agent_port: 4040,
      grpc_port: 9040
    ]
  ]

config :my_app, MyApp.PromExBiz,
  grafana_agent: [
    working_directory: System.fetch_env!("RELEASE_TMP") <> "/grafana-biz",
    config_opts: [
      ...
      agent_port: 4041,
      grpc_port: 9041
    ]
  ]
```

--------------------------------

### Configure Grafana Agent for Metrics

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Configuring Grafana Agent.md

This configuration is used to set up the Grafana Agent for sending metrics. Ensure GRAFANA_HOST and GRAFANA_TOKEN environment variables are set. Pay attention to 'metrics_server_scheme' if your services are behind a proxy.

```elixir
config :my_app, MyApp.PromEx,
    manual_metrics_start_delay: :no_delay,
    drop_metrics_groups: [],
    grafana: [
      host: System.get_env("GRAFANA_HOST") || raise("GRAFANA_HOST is required"),
      auth_token: System.get_env("GRAFANA_TOKEN") || raise("GRAFANA_TOKEN is required"),
      upload_dashboards_on_start: true,
      folder_name: "MyApp Dashboards",
      annotate_app_lifecycle: true
    ],
    grafana_agent: [
      working_directory: System.get_env("PROM_WORKING_DIRECTORY"),
      config_opts: [
        prometheus_url: System.get_env("PROM_URL"),
        prometheus_username: System.get_env("PROM_USER"),
        prometheus_password: System.get_env("PROM_PASS"),
        metrics_server_scheme: :http
      ]
    ]
```

--------------------------------

### Configure Shared Endpoint for PromEx Agents

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Running Multiple Agents.md

Configure a shared endpoint for multiple PromEx agents to scrape the same set of metrics. Ensure the `metrics_server_path` is set in your mix config.

```elixir
# endpoint.ex
plug PromEx.Plug, prom_ex_module: MyApp.PromExOps

# in mix config, set `metrics_server_path: "/metrics"`
```

--------------------------------

### ApacheBench Stress Test without PromEx

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Results from an ApacheBench stress test showing performance metrics when PromEx is not active. This highlights the performance overhead, if any, introduced by PromEx.

```terminal
$ ./benchmarks/ab-graph.sh -u http://localhost:4000 -n 1000 -c 50 -k
Server Software:        Cowboy
Server Hostname:        localhost
Server Port:            4000

Document Path:          /
Document Length:        3389 bytes

Concurrency Level:      50
Time taken for tests:   4.156 seconds
Complete requests:      1000
Failed requests:        0
Keep-Alive requests:    1000
Total transferred:      4060000 bytes
HTML transferred:       3389000 bytes
Requests per second:    240.59 [#/sec] (mean)
Time per request:       207.822 [ms] (mean)
Time per request:       4.156 [ms] (mean, across all concurrent requests)
Transfer rate:          953.90 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    38  202  23.1    205     267
Waiting:       37  202  23.1    205     267
Total:         38  202  23.0    205     267

Percentage of the requests served within a certain time (ms)
  50%    205
  66%    211
  75%    215
  80%    219
  90%    226
  95%    232
  98%    238
  99%    246
 100%    267 (longest request)

```

--------------------------------

### Add PromEx Plug to Endpoint

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Integrate the PromEx plug into your Phoenix endpoint to expose metrics. It's recommended to place it before `Plug.Telemetry`.

```elixir
defmodule MyCoolAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :my_cool_app

  ...

  plug PromEx.Plug, prom_ex_module: MyCoolApp.PromEx
  # Or plug PromEx.Plug, path: "/some/other/metrics/path", prom_ex_module: MyCoolApp.PromEx

  ...

  plug Plug.RequestId
  plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]

  ...

  plug MyCoolAppWeb.Router
end
```

--------------------------------

### Attach Telemetry Callbacks for Event Handling

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Telemetry.md

This snippet demonstrates how to attach callback functions to specific Telemetry events using `:telemetry.attach/4`. It sets up handlers for both success and error events related to user registration, logging different messages based on the event outcome.

```elixir
:telemetry.attach(
  "my-handler-1",
  [:my_cool_app, :accounts, :new_user, :success],
  fn _event_name, _event_measurement, event_metadata, _config ->
    Logger.debug("User has registered: #{inspect(event_metadata)}")
  end,
  %{})

:telemetry.attach(
  "my-handler-2",
  [:my_cool_app, :accounts, :new_user, :error],
  fn _event_name, _event_measurement, event_metadata, _config ->
    Logger.warning("User failed to register: #{inspect(event_metadata)}")
  end,
  %{})

```

--------------------------------

### Seed Event Metrics with Telemetry

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Seeding Metrics.md

Define the `seed_event_metrics/0` function to emit various telemetry events, ensuring the metrics server path is populated.

```elixir
def seed_event_metrics do
  # :telemetry.execute([:my_app, :event], %{}, %{status: :ok})
  # :telemetry.execute([:my_app, :event], %{}, %{status: :error})
  # :telemetry.execute([:my_app, :event_span, :start], %{count: 1}, %{})
  # :telemetry.execute([:my_app, :event_span, :stop], %{count: 1}, %{})
  # :telemetry.execute([:my_app, :event_span, :exception], %{count: 1}, %{})

  :ok
end
```

--------------------------------

### Add PromEx to Mix Dependencies

Source: https://github.com/akoutmos/prom_ex/blob/master/README.md

Add PromEx to your project's dependencies in `mix.exs` to include it in your Elixir application.

```elixir
def deps do
  [
    {:prom_ex, "~> 1.11.0"}
  ]
end
```

--------------------------------

### Emit Telemetry Events for User Registration

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Telemetry.md

This snippet shows how to emit Telemetry events for both successful user registration and errors within a registration function. It uses `:telemetry.execute/3` to send event names, measurements, and metadata.

```elixir
def register_user(attrs) do
  %User{}
  |> User.registration_changeset(updated_attrs)
  |> Repo.insert()
  |> case do
    {:ok, new_user} = result ->
      :telemetry.execute([:my_cool_app, :accounts, :new_user, :success], %{}, %{user: new_user})
      result

    {:error, changeset} = error ->
      :telemetry.execute([:my_cool_app, :accounts, :new_user, :error], %{}, %{error: changeset})
      error
  end
end
```

--------------------------------

### Define Polling Metrics in a PromEx Plugin

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Writing PromEx Plugins.md

Implement the `polling_metrics/1` callback to define metrics that are polled at a specified interval. The `measurements_mfa` argument in `Polling.build/4` specifies the function to execute for collecting measurements.

```elixir
defmodule PromEx.Plugins.Beam do
  use PromEx.Plugin

  @memory_event [:prom_ex, :plugin, :beam, :memory]

  @impl true
  def polling_metrics(opts) do
    poll_rate = Keyword.get(opts, :poll_rate, 5_000)

    [
      memory_metrics(poll_rate)
    ]
  end

  defp memory_metrics(poll_rate) do
    Polling.build(
      :beam_memory_polling_events,
      poll_rate,
      {__MODULE__, :execute_memory_metrics, []},
      [
        # Capture the total memory allocated to the entire Erlang VM (or BEAM for short)
        last_value(
          [:beam, :memory, :total, :kilobytes],
          event_name: @memory_event,
          description: "The total amount of memory currently allocated.",
          measurement: :total,
          unit: {:byte, :kilobyte}
        )

        # More memory metrics here
      ]
    )
  end

  @doc false
  def execute_memory_metrics do
    memory_measurements =
      :erlang.memory()
      |> Map.new()

    :telemetry.execute(@memory_event, memory_measurements, %{})
  end
end
```

--------------------------------

### Configure Separate Endpoints for PromEx Agents

Source: https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Running Multiple Agents.md

Define separate endpoints for each PromEx agent to scrape distinct sets of metrics. This is useful when agents serve different purposes or teams.

```elixir
plug PromEx.Plug, prom_ex_module: MyApp.PromExOps, path: "/metrics/ops"
plug PromEx.Plug, prom_ex_module: MyApp.PromExBiz, path: "/metrics/biz"
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.