VictoriaTraces

https://docs.victoriametrics.com/victoriatraces/
Admin
VictoriaTraces is an open-source, resource-efficient database for storing and querying distributed...
Tokens:50,440
Snippets:386
Trust Score:10
Update:9 months ago
Show doc for...
Tokens:
Raw
### Run HotROD Example Application for Tracing

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

This command starts the Jaeger HotROD example application in a Docker container. It configures the application to export traces to VictoriaTraces via OTLP/HTTP by setting the `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable. Users can then interact with the application to generate trace data.

```docker
docker run \
  -p8080-8083:8080-8083 \
  --rm \
  --env OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<victoria-traces>:10428/insert/opentelemetry/v1/traces \
  jaegertracing/example-hotrod:latest \
  all
```

--------------------------------

### Extract and Run VictoriaTraces Binary

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Extracts the downloaded VictoriaTraces archive and then starts the production-ready binary. This allows running VictoriaTraces directly on the host system.

```bash
tar -xvf victoria-traces-linux-amd64-v0.6.0.tar.gz
./victoria-traces-prod
```

--------------------------------

### Start VictoriaTraces Cluster Components

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Starts the vtstorage, vtinsert, and vtselect nodes for a VictoriaTraces cluster. Each command starts a component as a background process.

```bash
# Start the first vtstorage node
./victoria-traces-prod -httpListenAddr=:10491 -storageDataPath=victoria-traces-data-1 &

# Start the second vtstorage node
./victoria-traces-prod -httpListenAddr=:10492 -storageDataPath=victoria-traces-data-2 &

# Start vtinsert node
./victoria-traces-prod -httpListenAddr=:10481 -storageNode=localhost:10491,localhost:10492 &

# Start vtselect node
./victoria-traces-prod -httpListenAddr=:10471 -storageNode=localhost:10491,localhost:10492 &
```

--------------------------------

### Run VictoriaTraces Single Node via Docker

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Starts the latest VictoriaTraces release using a Docker container. This command maps port 10428 and mounts a volume for data persistence. It runs in the foreground and outputs logs to the console.

```docker
docker run --rm -it -p 10428:10428 -v ./victoria-traces-data:/victoria-traces-data \
  docker.io/victoriametrics/victoria-traces:latest
```

--------------------------------

### Download VictoriaTraces Binary Release

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Downloads the VictoriaTraces binary for Linux amd64 architecture using curl. This is a prerequisite for running VictoriaTraces from a binary distribution.

```bash
curl -L -O https://github.com/VictoriaMetrics/VictoriaTraces/releases/download/v0.6.0/victoria-traces-linux-amd64-v0.6.0.tar.gz
```

--------------------------------

### vmalert Recording Rule Examples for VictoriaTraces (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

Examples of recording rule configurations for vmalert using LogsQL, compatible with VictoriaTraces. These examples demonstrate how to record metrics like 'nginxRequestCount' and 'prodRequestCount' based on resource attributes and environment.

```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
```

--------------------------------

### Build VictoriaTraces Binary with Go (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Builds the VictoriaTraces binary using the Go toolchain. Requires Go to be installed. Places the executable in the 'bin' folder.

```bash
make victoria-traces
```

--------------------------------

### Build VictoriaTraces Binary with Docker (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Builds the VictoriaTraces production-ready binary using Docker. Requires Docker to be installed. Places the executable in the 'bin' folder.

```bash
make victoria-traces-prod
```

--------------------------------

### VictoriaTraces Recording Rule Example (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of recording rules using LogsQL for VictoriaTraces. It demonstrates recording 'nginxRequestCount' based on 'test' environment and service name 'nginx', and 'prodRequestCount' for the 'prod' environment.

```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
    

```

--------------------------------

### Query Dependencies with Time Range (curl)

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This example demonstrates how to query for service dependencies within a specified time range using the `curl` command. It utilizes the `endTs` and `lookback` parameters to define the query window. Ensure VictoriaTraces is configured with the experimental service graph feature enabled.

```curl
curl http://<victoria-traces>:10428/select/jaeger/api/dependencies?endTs=1758213428616&lookback=60000
```

--------------------------------

### Get VictoriaTraces Help Information (Shell)

Source: https://docs.victoriametrics.com/victoriatraces/index

Displays all supported command-line flags for VictoriaTraces, including their descriptions and default values. Useful for configuration.

```shell
/path/to/victoria-traces -help
```

--------------------------------

### Client SDK - HTTP Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry HTTP exporter to send trace data to VictoriaTraces.

```APIDOC
## POST /insert/opentelemetry/v1/traces

### Description
Send trace data to VictoriaTraces via the OpenTelemetry HTTP endpoint.

### Method
POST

### Endpoint
`http://<victoria-traces>:10428/insert/opentelemetry/v1/traces`

### Parameters
#### Query Parameters
None

#### Request Body
OpenTelemetry trace data in OTLP/HTTP format.

### Request Example (Go SDK)
```go
traceExporter, err := otlptracehttp.New(ctx,
    otlptracehttp.WithEndpointURL("http://<victoria-traces>:10428/insert/opentelemetry/v1/traces"),
)
```

### Response
#### Success Response (200)
Details on success response are not specified. Typically, a 2xx status code indicates successful ingestion.

#### Response Example
```json
{
  "message": "Traces received successfully"
}
```
```

--------------------------------

### Client SDK - gRPC Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry gRPC exporter to send trace data to VictoriaTraces.

```APIDOC
## POST /insert/opentelemetry/v1/traces (gRPC)

### Description
Send trace data to VictoriaTraces via the OpenTelemetry gRPC endpoint.

### Method
POST

### Endpoint
`<victoria-traces>:4317`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
OpenTelemetry trace data in OTLP/gRPC format.

### Server Configuration
To enable the OTLP gRPC server on VictoriaTraces:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tlsCertFile=<cert_file> -otlpGRPC.tlsKeyFile=<key_file>
```
To disable TLS:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
```

### Request Example (Go SDK with TLS)
```go
traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("<victoria-traces>:4317"),
    otlptracegrpc.WithTLSCredentials(<creds>),
)
```

### Request Example (Go SDK without TLS)
```go
traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("<victoria-traces>:4317"),
    otlptracegrpc.WithInsecure(),  // use insecure connection if TLS is disabled.
)
```

### Response
#### Success Response (200)
Details on success response are not specified. Typically, a 2xx status code indicates successful ingestion.

#### Response Example
```json
{
  "message": "Traces received successfully"
}
```
```

--------------------------------

### Recording Rules with VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example configuration for defining recording rules in `vmalert` when using VictoriaTraces, leveraging LogsQL for metric generation.

```APIDOC
## Recording Rules with VictoriaTraces

### Description
Defines how to set up recording rules within `vmalert` that query VictoriaTraces using LogsQL. These rules generate new time series based on log data, which can then be used for further analysis or alerting.

### Method
N/A (Rule configuration)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Rule Configuration Example
```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
```

### Response
N/A (Generated metrics are persisted via `-remoteWrite.url`.
```

--------------------------------

### Configure vmauth for Routing VictoriaTraces and VictoriaMetrics Queries

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

This vmauth configuration example shows how to route incoming requests based on their source paths. Requests matching '/api/v1/query.*' are sent to VictoriaMetrics, while requests matching '/select/logsql/.*' are routed to VictoriaTraces.

```yaml
unauthorized_user:
  url_map:
    - src_paths:
      - "/api/v1/query.*"
      url_prefix: "http://victoriametrics:8428"
    - src_paths:
      - "/select/logsql/.*"
      url_prefix: "http://victoriatraces:10428"
```

--------------------------------

### Collector Configuration - gRPC Exporter

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry Collector's OTLP/gRPC exporter to send traces to VictoriaTraces.

```APIDOC
## OpenTelemetry Collector - OTLP/gRPC Exporter

### Description
Configure the OpenTelemetry Collector to export collected traces to VictoriaTraces using the OTLP/gRPC protocol.

### Server Configuration on VictoriaTraces
Ensure the OTLP gRPC server is enabled on VictoriaTraces:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tlsCertFile=<cert_file> -otlpGRPC.tlsKeyFile=<key_file>
```
To disable TLS:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
```

### Configuration File Snippet (with TLS)

```yaml
exporters:
  otlp/with-tls:
    endpoint: <victoria-traces>:4317
    tls:
      cert_file: file.cert
      key_file: file.key
```

### Configuration File Snippet (without TLS)

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
```

### Adding Custom Headers
Custom HTTP headers can be added to the exporter configuration.

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
    headers:
      VT-Extra-Fields: foo=bar
```

### Supported Compression
Compression can be specified using the `compression` field (e.g., `gzip`, `snappy`, `zstd`, `none`). Default is `gzip`.

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
    compression: zstd
```
```

--------------------------------

### Collector Configuration - HTTP Exporter

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry Collector's OTLP/HTTP exporter to send traces to VictoriaTraces.

```APIDOC
## OpenTelemetry Collector - OTLP/HTTP Exporter

### Description
Configure the OpenTelemetry Collector to export collected traces to VictoriaTraces using the OTLP/HTTP protocol.

### Configuration File Snippet

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
```

### Adding Custom Headers
Custom HTTP headers can be added to the exporter configuration.

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
    headers:
      VT-Extra-Fields: foo=bar
```

### Supported Compression
Compression can be specified using the `compression` field (e.g., `gzip`, `snappy`, `zstd`, `none`). Default is `gzip`.

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
    compression: snappy
```
```

--------------------------------

### Run vmalert with VictoriaTraces Configuration

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

Command to run vmalert with parameters for integrating with VictoriaTraces, AlertManager, and VictoriaMetrics. This setup allows vmalert to query trace span stats from VictoriaTraces, send notifications to AlertManager, and persist rule states.

```bash
./bin/vmalert -rule=alert.rules                  \
    -datasource.url=http://victoriatraces:10428  \
    -notifier.url=http://alertmanager:9093       \
    -remoteWrite.url=http://victoriametrics:8428 \
    -remoteRead.url=http://victoriametrics:8428
```

--------------------------------

### List Services using VictoriaTraces API

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This command retrieves a list of all available services tracked by VictoriaTraces. It makes a GET request to the `/select/jaeger/api/services` endpoint. The response is a JSON object containing an array of service names.

```shell
curl http://<victoria-traces>:10428/select/jaeger/api/services
```

--------------------------------

### vmalert Group Configuration for VictoriaTraces Multitenancy (Method 1)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of vmalert group configurations for handling multitenancy with VictoriaTraces and persisting to VictoriaMetrics. It shows how `headers` and `labels` are used to define the queried tenant from VictoriaTraces and the target tenant in VictoriaMetrics.

```yaml
groups:
  - name: recordingTenant123
    type: vlogs
    headers:
      - "AccountID: 123"
    labels:
      vm_account_id: 123
    rules:
      - record: recordingTenant123
        expr: 'tags.path:/var/log/httpd OR tags.path:/var/log/nginx | stats by (tags.host) count() requests'
  - name: recordingTenant124-456:789
    type: vlogs
    headers:
      - "AccountID: 124"
    labels:
      vm_account_id: 456
      vm_project_id: 789
    rules:
    - record: recordingTenant124-456:789
        expr: 'tags.path:/var/log/httpd OR tags.path:/var/log/nginx | stats by (tags.host) count() requests'

```

--------------------------------

### Monitoring Metrics

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Access internal metrics exposed by VictoriaTraces in Prometheus format for monitoring purposes.

```APIDOC
## GET /metrics

### Description
This endpoint exposes VictoriaTraces' internal metrics in Prometheus exposition format. It is recommended to scrape these metrics using Prometheus or vmagent for monitoring.

### Method
GET

### Endpoint
`/metrics`

### Parameters
None

### Response
#### Success Response (200)
- **metrics** (string) - A text-based output of metrics in Prometheus format.

#### Response Example
```text
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 25
# HELP http_requests_total Total number of HTTP requests received.
# TYPE http_requests_total counter
http_requests_total{code="200",method="POST",path="/insert/opentelemetry/v1/traces"} 150
```
```

--------------------------------

### Write Data (OTLP/HTTP)

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Ingest trace spans into VictoriaTraces using the OpenTelemetry Protocol (OTLP) over HTTP.

```APIDOC
## POST /insert/opentelemetry/v1/traces

### Description
This endpoint accepts trace spans in OpenTelemetry Protocol (OTLP) format via HTTP.

### Method
POST

### Endpoint
`/insert/opentelemetry/v1/traces`

### Parameters
#### Request Body
- **resourceSpans** (array) - Required - An array of resource spans conforming to the OTLP schema.

### Request Example
```json
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {"stringValue": "frontend-web"}
          },
          {
            "key": "telemetry.sdk.language",
            "value": {"stringValue": "webjs"}
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {"name": "@opentelemetry/instrumentation-document-load", "version": "0.44.1"},
          "spans": [
            {
              "traceId": "1af5dd013a30efe7f2970032ab81958b",
              "spanId": "229d083a6c480511",
              "name": "documentLoad",
              "kind": 1,
              "startTimeUnixNano": "ingestTimePlaceHolder",
              "endTimeUnixNano": "ingestTimePlaceHolder",
              "attributes": [
                {
                  "key": "session.id",
                  "value": {"stringValue": "96e702c3-6f05-4f54-b2b3-2fad2b7b7995"}
                },
                {
                  "key": "http.url",
                  "value": {"stringValue": "http://frontend-proxy:8080/cart"}
                }
              ],
              "events": [
                {
                  "timeUnixNano": "1757320936519100098",
                  "name": "fetchStart"
                }
              ],
              "status": {}
            }
          ]
        }
      ]
    }
  ]
}
```

### Response
#### Success Response (200)
- **message** (string) - A success message indicating data ingestion.

#### Response Example
```json
{
  "message": "Traces ingested successfully"
}
```
```

--------------------------------

### Query Data (Jaeger)

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Access trace data through Jaeger Query Service JSON APIs for visualization in tools like Grafana.

```APIDOC
## GET /select/jaeger

### Description
This endpoint provides access to trace data via Jaeger Query Service JSON APIs. It is useful for integrating with systems like Grafana.

### Method
GET

### Endpoint
`/select/jaeger`

### Parameters
#### Query Parameters
- **service** (string) - Optional - Filter traces by service name.
- **operation** (string) - Optional - Filter traces by operation name.
- **limit** (integer) - Optional - The maximum number of traces to return.
- **start** (integer) - Optional - Start time for the query in milliseconds since epoch.
- **end** (integer) - Optional - End time for the query in milliseconds since epoch.

### Response
#### Success Response (200)
- **data** (object) - Contains trace data in Jaeger format.
  - **traces** (array) - An array of trace objects.

#### Response Example
```json
{
  "data": {
    "traces": [
      {
        "traceID": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
        "spans": [
          {
            "traceID": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
            "spanID": "1234567890abcdef",
            "operationName": "http.request",
            "startTime": 1678886400000000000,
            "duration": 1000000,
            "tags": [
              {"key": "http.method", "type": "string", "value": "GET"},
              {"key": "http.url", "type": "string", "value": "/api/users"}
            ]
          }
        ]
      }
    ]
  }
}
```
```

--------------------------------

### Enable Basic Auth for vtinsert and vtselect clients

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtinsert and vtselect to use Basic Authentication when communicating with vtstorage. Set the `-storageNode.username` and `-storageNode.password` flags on the client components for secure authentication over untrusted networks.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls -storageNode.username=... -storageNode.password=...
```

--------------------------------

### Enable TLS for vtinsert and vtselect clients

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtinsert and vtselect to communicate with vtstorage over TLS by setting the `-storageNode.tls` command-line flag. This ensures encrypted data transfer when these components interact over untrusted networks.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls
```

--------------------------------

### Expose Internal Metrics in Prometheus Format

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

VictoriaTraces exposes its internal metrics in Prometheus exposition format. This endpoint can be scraped by Prometheus or other monitoring solutions to track the health and performance of VictoriaTraces itself.

```http
http://<victoria-traces>:10428/metrics
```

--------------------------------

### Jaeger Query Service API Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

VictoriaTraces exposes trace data through Jaeger Query Service compatible APIs. This URL can be used to configure a Jaeger datasource in tools like Grafana for visualizing trace data.

```http
http://<victoria-traces>:10428/select/jaeger
```

--------------------------------

### Enable TLS on vtstorage

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtstorage to accept incoming requests over HTTPS by specifying the `-tls`, `-tlsCertFile`, and `-tlsKeyFile` command-line flags. This is essential for secure communication over untrusted networks.

```bash
./victoria-traces-prod -httpListenAddr=... -storageDataPath=... -tls -tlsCertFile=/path/to/certfile -tlsKeyFile=/path/to/keyfile
```

--------------------------------

### Ingest Trace Data via OTLP/HTTP using curl

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

This snippet demonstrates how to send a trace span to VictoriaTraces using a curl command. It constructs a JSON payload representing a trace span and sends it to the `/insert/opentelemetry/v1/traces` endpoint. The `ingestTimePlaceHolder` is dynamically replaced with the current time.

```shell
echo '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"frontend-web"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.30.1"}},{"key":"process.runtime.name","value":{"stringValue":"browser"}},{"key":"process.runtime.description","value":{"stringValue":"Web Browser"}},{"key":"process.runtime.version","value":{"stringValue":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.0.0 Safari/537.36"}}]},"scopeSpans":[{"scope":{"name":"@opentelemetry/instrumentation-document-load","version":"0.44.1"},"spans":[{"traceId":"1af5dd013a30efe7f2970032ab81958b","spanId":"229d083a6c480511","parentSpanId":"","name":"documentLoad","kind":1,"startTimeUnixNano":"ingestTimePlaceHolder","endTimeUnixNano":"ingestTimePlaceHolder","attributes":[{"key":"session.id","value":{"stringValue":"96e702c3-6f05-4f54-b2b3-2fad2b7b7995"}},{"key":"http.url","value":{"stringValue":"http://frontend-proxy:8080/cart"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.0.0 Safari/537.36"}}],"events":[{"timeUnixNano":"1757320936519100098","name":"fetchStart"}],"status":{}}]}]}]}' |
sed "s/ingestTimePlaceHolder/$(date +%s000000000)/g" |
curl -X POST -H 'Content-Type: application/json' --data-binary @- http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
```

--------------------------------

### Download and Unpack VictoriaTraces Executable

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Downloads the latest VictoriaTraces release for Linux amd64 and unpacks the archive. This is the initial step before setting up the cluster.

```bash
curl -L -O https://github.com/VictoriaMetrics/VictoriaTraces/releases/download/v0.6.0/victoria-traces-linux-amd64-v0.6.0.tar.gz
tar xzf victoria-traces-linux-amd64-v0.6.0.tar.gz
```

--------------------------------

### Run vmalert with VictoriaTraces Datasource

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to run vmalert, specifying the path to rule files, the VictoriaTraces datasource URL, the AlertManager URL, and remote write/read URLs for state persistence and restoration.

```bash
./bin/vmalert -rule=alert.rules                  \
    -datasource.url=http://victoriatraces:10428  \
    -notifier.url=http://alertmanager:9093       \
    -remoteWrite.url=http://victoriametrics:8428 \
    -remoteRead.url=http://victoriametrics:8428  \
    

```

--------------------------------

### VictoriaTraces Alerting Rule Example (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of an alerting rule group using LogsQL for VictoriaTraces. It defines an alert 'TooManySpans' that triggers if more than 50 spans are received for a specific span name within 5 minutes.

```yaml
groups:
  - name: CountBySpanName
    # The use of `type: vlogs` is because VictoriaTraces uses LogsQL (the same query language as VictoriaLogs).
    # No extra VictoriaLogs setup is needed during the process.
    type: vlogs
    interval: 5m
    rules:
       - alert: TooManySpans
         expr: '* | stats by (name) count() as total | filter total:>50'
         annotations:
            description: "Received more than 50 trace spans for span name {{$labels.name}} in the last 5 minute: {{$value}}"
    

```

--------------------------------

### vmalert Alerting Rule Example for VictoriaTraces (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

An example of an alerting rule configuration for vmalert using LogsQL, suitable for VictoriaTraces. This defines a rule named 'TooManySpans' that triggers when more than 50 spans are received for a span name in the last 5 minutes.

```yaml
groups:
  - name: CountBySpanName
    type: vlogs
    interval: 5m
    rules:
       - alert: TooManySpans
         expr: '* | stats by (name) count() as total | filter total:>50'
         annotations:
            description: "Received more than 50 trace spans for span name {{$labels.name}} in the last 5 minute: {{$value}}"
```

--------------------------------

### Configure client TLS certificates for mTLS

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Specify client TLS certificates for vtinsert and vtselect nodes when using mTLS with vtstorage. Use the `-storageNode.tlsCertFile` and `-storageNode.tlsKeyFile` flags.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls -storageNode.tlsCertFile=/path/to/client.cert -storageNode.tlsKeyFile=/path/to/client.key
```

--------------------------------

### Run Enterprise vmalert with Cluster Mode for Multitenancy

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to run the enterprise version of vmalert with cluster mode enabled. This approach allows specifying the tenant directly per group configuration, facilitating multitenancy for VictoriaTraces.

```bash
./bin/vmalert -datasource.url=http://localhost:9428 -clusterMode=true -remoteWrite.url=http://vminsert:8480/ ...

```

--------------------------------

### vmalert Rule Backfilling Command

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to perform rule backfilling (replay) against VictoriaTraces using vmalert. It specifies the path to the rules file, the VictoriaTraces datasource URL, the default rule type, a remote write URL for state persistence, and the time range for the replay.

```bash
./bin/vmalert -rule=path/to/your.rules \
    -datasource.url=http://localhost:10428 \
    -rule.defaultRuleType=vlogs \
    -remoteWrite.url=http://localhost:8428 \
    -replay.timeFrom=2021-05-11T07:21:43Z \
    -replay.timeTo=2021-05-29T18:40:43Z
```
Tokens:
Raw