### Run HotROD Example Application for Tracing

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

This command starts the Jaeger HotROD example application in a Docker container. It configures the application to export traces to VictoriaTraces via OTLP/HTTP by setting the `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable. Users can then interact with the application to generate trace data.

```docker
docker run \
  -p8080-8083:8080-8083 \
  --rm \
  --env OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<victoria-traces>:10428/insert/opentelemetry/v1/traces \
  jaegertracing/example-hotrod:latest \
  all
```

--------------------------------

### Extract and Run VictoriaTraces Binary

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Extracts the downloaded VictoriaTraces archive and then starts the production-ready binary. This allows running VictoriaTraces directly on the host system.

```bash
tar -xvf victoria-traces-linux-amd64-v0.6.0.tar.gz
./victoria-traces-prod
```

--------------------------------

### Start VictoriaTraces Cluster Components

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Starts the vtstorage, vtinsert, and vtselect nodes for a VictoriaTraces cluster. Each command starts a component as a background process.

```bash
# Start the first vtstorage node
./victoria-traces-prod -httpListenAddr=:10491 -storageDataPath=victoria-traces-data-1 &

# Start the second vtstorage node
./victoria-traces-prod -httpListenAddr=:10492 -storageDataPath=victoria-traces-data-2 &

# Start vtinsert node
./victoria-traces-prod -httpListenAddr=:10481 -storageNode=localhost:10491,localhost:10492 &

# Start vtselect node
./victoria-traces-prod -httpListenAddr=:10471 -storageNode=localhost:10491,localhost:10492 &
```

--------------------------------

### Run VictoriaTraces Single Node via Docker

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Starts the latest VictoriaTraces release using a Docker container. This command maps port 10428 and mounts a volume for data persistence. It runs in the foreground and outputs logs to the console.

```docker
docker run --rm -it -p 10428:10428 -v ./victoria-traces-data:/victoria-traces-data \
  docker.io/victoriametrics/victoria-traces:latest
```

--------------------------------

### Download VictoriaTraces Binary Release

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Downloads the VictoriaTraces binary for Linux amd64 architecture using curl. This is a prerequisite for running VictoriaTraces from a binary distribution.

```bash
curl -L -O https://github.com/VictoriaMetrics/VictoriaTraces/releases/download/v0.6.0/victoria-traces-linux-amd64-v0.6.0.tar.gz
```

--------------------------------

### vmalert Recording Rule Examples for VictoriaTraces (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

Examples of recording rule configurations for vmalert using LogsQL, compatible with VictoriaTraces. These examples demonstrate how to record metrics like 'nginxRequestCount' and 'prodRequestCount' based on resource attributes and environment.

```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
```

--------------------------------

### Build VictoriaTraces Binary with Go (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Builds the VictoriaTraces binary using the Go toolchain. Requires Go to be installed. Places the executable in the 'bin' folder.

```bash
make victoria-traces
```

--------------------------------

### Build VictoriaTraces Binary with Docker (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Builds the VictoriaTraces production-ready binary using Docker. Requires Docker to be installed. Places the executable in the 'bin' folder.

```bash
make victoria-traces-prod
```

--------------------------------

### VictoriaTraces Recording Rule Example (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of recording rules using LogsQL for VictoriaTraces. It demonstrates recording 'nginxRequestCount' based on 'test' environment and service name 'nginx', and 'prodRequestCount' for the 'prod' environment.

```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
    

```

--------------------------------

### Query Dependencies with Time Range (curl)

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This example demonstrates how to query for service dependencies within a specified time range using the `curl` command. It utilizes the `endTs` and `lookback` parameters to define the query window. Ensure VictoriaTraces is configured with the experimental service graph feature enabled.

```curl
curl http://<victoria-traces>:10428/select/jaeger/api/dependencies?endTs=1758213428616&lookback=60000
```

--------------------------------

### Monitoring Metrics

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Access internal metrics exposed by VictoriaTraces in Prometheus format for monitoring purposes.

```APIDOC
## GET /metrics

### Description
This endpoint exposes VictoriaTraces' internal metrics in Prometheus exposition format. It is recommended to scrape these metrics using Prometheus or vmagent for monitoring.

### Method
GET

### Endpoint
`/metrics`

### Parameters
None

### Response
#### Success Response (200)
- **metrics** (string) - A text-based output of metrics in Prometheus format.

#### Response Example
```text
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 25
# HELP http_requests_total Total number of HTTP requests received.
# TYPE http_requests_total counter
http_requests_total{code="200",method="POST",path="/insert/opentelemetry/v1/traces"} 150
```
```

--------------------------------

### Get VictoriaTraces Help Information (Shell)

Source: https://docs.victoriametrics.com/victoriatraces/index

Displays all supported command-line flags for VictoriaTraces, including their descriptions and default values. Useful for configuration.

```shell
/path/to/victoria-traces -help
```

--------------------------------

### Write Data (OTLP/HTTP)

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Ingest trace spans into VictoriaTraces using the OpenTelemetry Protocol (OTLP) over HTTP.

```APIDOC
## POST /insert/opentelemetry/v1/traces

### Description
This endpoint accepts trace spans in OpenTelemetry Protocol (OTLP) format via HTTP.

### Method
POST

### Endpoint
`/insert/opentelemetry/v1/traces`

### Parameters
#### Request Body
- **resourceSpans** (array) - Required - An array of resource spans conforming to the OTLP schema.

### Request Example
```json
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {"stringValue": "frontend-web"}
          },
          {
            "key": "telemetry.sdk.language",
            "value": {"stringValue": "webjs"}
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {"name": "@opentelemetry/instrumentation-document-load", "version": "0.44.1"},
          "spans": [
            {
              "traceId": "1af5dd013a30efe7f2970032ab81958b",
              "spanId": "229d083a6c480511",
              "name": "documentLoad",
              "kind": 1,
              "startTimeUnixNano": "ingestTimePlaceHolder",
              "endTimeUnixNano": "ingestTimePlaceHolder",
              "attributes": [
                {
                  "key": "session.id",
                  "value": {"stringValue": "96e702c3-6f05-4f54-b2b3-2fad2b7b7995"}
                },
                {
                  "key": "http.url",
                  "value": {"stringValue": "http://frontend-proxy:8080/cart"}
                }
              ],
              "events": [
                {
                  "timeUnixNano": "1757320936519100098",
                  "name": "fetchStart"
                }
              ],
              "status": {}
            }
          ]
        }
      ]
    }
  ]
}
```

### Response
#### Success Response (200)
- **message** (string) - A success message indicating data ingestion.

#### Response Example
```json
{
  "message": "Traces ingested successfully"
}
```
```

--------------------------------

### Client SDK - HTTP Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry HTTP exporter to send trace data to VictoriaTraces.

```APIDOC
## POST /insert/opentelemetry/v1/traces

### Description
Send trace data to VictoriaTraces via the OpenTelemetry HTTP endpoint.

### Method
POST

### Endpoint
`http://<victoria-traces>:10428/insert/opentelemetry/v1/traces`

### Parameters
#### Query Parameters
None

#### Request Body
OpenTelemetry trace data in OTLP/HTTP format.

### Request Example (Go SDK)
```go
traceExporter, err := otlptracehttp.New(ctx,
    otlptracehttp.WithEndpointURL("http://<victoria-traces>:10428/insert/opentelemetry/v1/traces"),
)
```

### Response
#### Success Response (200)
Details on success response are not specified. Typically, a 2xx status code indicates successful ingestion.

#### Response Example
```json
{
  "message": "Traces received successfully"
}
```
```

--------------------------------

### Client SDK - gRPC Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry gRPC exporter to send trace data to VictoriaTraces.

```APIDOC
## POST /insert/opentelemetry/v1/traces (gRPC)

### Description
Send trace data to VictoriaTraces via the OpenTelemetry gRPC endpoint.

### Method
POST

### Endpoint
`<victoria-traces>:4317`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
OpenTelemetry trace data in OTLP/gRPC format.

### Server Configuration
To enable the OTLP gRPC server on VictoriaTraces:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tlsCertFile=<cert_file> -otlpGRPC.tlsKeyFile=<key_file>
```
To disable TLS:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
```

### Request Example (Go SDK with TLS)
```go
traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("<victoria-traces>:4317"),
    otlptracegrpc.WithTLSCredentials(<creds>),
)
```

### Request Example (Go SDK without TLS)
```go
traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("<victoria-traces>:4317"),
    otlptracegrpc.WithInsecure(),  // use insecure connection if TLS is disabled.
)
```

### Response
#### Success Response (200)
Details on success response are not specified. Typically, a 2xx status code indicates successful ingestion.

#### Response Example
```json
{
  "message": "Traces received successfully"
}
```
```

--------------------------------

### Recording Rules with VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example configuration for defining recording rules in `vmalert` when using VictoriaTraces, leveraging LogsQL for metric generation.

```APIDOC
## Recording Rules with VictoriaTraces

### Description
Defines how to set up recording rules within `vmalert` that query VictoriaTraces using LogsQL. These rules generate new time series based on log data, which can then be used for further analysis or alerting.

### Method
N/A (Rule configuration)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Rule Configuration Example
```yaml
groups:
  - name: SpanCount
    type: vlogs
    interval: 5m
    rules:
      - record: nginxRequestCount
        expr: 'env: "test" AND resource_attr:service.name: "nginx" | stats count(*) as requests'
      - record: prodRequestCount
        expr: 'env: "prod" | stats by (resource_attr:service.name) count(*) as requests'
```

### Response
N/A (Generated metrics are persisted via `-remoteWrite.url`.
```

--------------------------------

### Configure vmauth for Routing VictoriaTraces and VictoriaMetrics Queries

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

This vmauth configuration example shows how to route incoming requests based on their source paths. Requests matching '/api/v1/query.*' are sent to VictoriaMetrics, while requests matching '/select/logsql/.*' are routed to VictoriaTraces.

```yaml
unauthorized_user:
  url_map:
    - src_paths:
      - "/api/v1/query.*"
      url_prefix: "http://victoriametrics:8428"
    - src_paths:
      - "/select/logsql/.*"
      url_prefix: "http://victoriatraces:10428"
```

--------------------------------

### Collector Configuration - gRPC Exporter

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry Collector's OTLP/gRPC exporter to send traces to VictoriaTraces.

```APIDOC
## OpenTelemetry Collector - OTLP/gRPC Exporter

### Description
Configure the OpenTelemetry Collector to export collected traces to VictoriaTraces using the OTLP/gRPC protocol.

### Server Configuration on VictoriaTraces
Ensure the OTLP gRPC server is enabled on VictoriaTraces:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tlsCertFile=<cert_file> -otlpGRPC.tlsKeyFile=<key_file>
```
To disable TLS:
```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
```

### Configuration File Snippet (with TLS)

```yaml
exporters:
  otlp/with-tls:
    endpoint: <victoria-traces>:4317
    tls:
      cert_file: file.cert
      key_file: file.key
```

### Configuration File Snippet (without TLS)

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
```

### Adding Custom Headers
Custom HTTP headers can be added to the exporter configuration.

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
    headers:
      VT-Extra-Fields: foo=bar
```

### Supported Compression
Compression can be specified using the `compression` field (e.g., `gzip`, `snappy`, `zstd`, `none`). Default is `gzip`.

```yaml
exporters:
  otlp/without-tls:
    endpoint: <victoria-traces>:4317
    tls:
      insecure: true
    compression: zstd
```
```

--------------------------------

### Collector Configuration - HTTP Exporter

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry/index

Configure the OpenTelemetry Collector's OTLP/HTTP exporter to send traces to VictoriaTraces.

```APIDOC
## OpenTelemetry Collector - OTLP/HTTP Exporter

### Description
Configure the OpenTelemetry Collector to export collected traces to VictoriaTraces using the OTLP/HTTP protocol.

### Configuration File Snippet

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
```

### Adding Custom Headers
Custom HTTP headers can be added to the exporter configuration.

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
    headers:
      VT-Extra-Fields: foo=bar
```

### Supported Compression
Compression can be specified using the `compression` field (e.g., `gzip`, `snappy`, `zstd`, `none`). Default is `gzip`.

```yaml
exporters:
  otlphttp:
    traces_endpoint: http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
    compression: snappy
```
```

--------------------------------

### Alerting Rules with VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example configuration for defining alerting rules in `vmalert` when using VictoriaTraces, utilizing LogsQL for expressions.

```APIDOC
## Alerting Rules with VictoriaTraces

### Description
Defines how to set up alerting rules within `vmalert` that query VictoriaTraces using LogsQL. This allows for real-time monitoring and notification based on log data patterns.

### Method
N/A (Rule configuration)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Rule Configuration Example
```yaml
groups:
  - name: CountBySpanName
    type: vlogs  # Use 'vlogs' for VictoriaTraces/LogsQL
    interval: 5m
    rules:
      - alert: TooManySpans
        expr: '* | stats by (name) count() as total | filter total:>50'
        annotations:
          description: "Received more than 50 trace spans for span name {{$labels.name}} in the last 5 minute: {{$value}}"
```

### Response
N/A (Rule evaluation results are sent to Alertmanager.
```

--------------------------------

### Run vmalert with VictoriaTraces Configuration

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

Command to run vmalert with parameters for integrating with VictoriaTraces, AlertManager, and VictoriaMetrics. This setup allows vmalert to query trace span stats from VictoriaTraces, send notifications to AlertManager, and persist rule states.

```bash
./bin/vmalert -rule=alert.rules                  \
    -datasource.url=http://victoriatraces:10428  \
    -notifier.url=http://alertmanager:9093       \
    -remoteWrite.url=http://victoriametrics:8428 \
    -remoteRead.url=http://victoriametrics:8428
```

--------------------------------

### List Services using VictoriaTraces API

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This command retrieves a list of all available services tracked by VictoriaTraces. It makes a GET request to the `/select/jaeger/api/services` endpoint. The response is a JSON object containing an array of service names.

```shell
curl http://<victoria-traces>:10428/select/jaeger/api/services
```

--------------------------------

### vmalert Group Configuration for VictoriaTraces Multitenancy (Method 1)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of vmalert group configurations for handling multitenancy with VictoriaTraces and persisting to VictoriaMetrics. It shows how `headers` and `labels` are used to define the queried tenant from VictoriaTraces and the target tenant in VictoriaMetrics.

```yaml
groups:
  - name: recordingTenant123
    type: vlogs
    headers:
      - "AccountID: 123"
    labels:
      vm_account_id: 123
    rules:
      - record: recordingTenant123
        expr: 'tags.path:/var/log/httpd OR tags.path:/var/log/nginx | stats by (tags.host) count() requests'
  - name: recordingTenant124-456:789
    type: vlogs
    headers:
      - "AccountID: 124"
    labels:
      vm_account_id: 456
      vm_project_id: 789
    rules:
    - record: recordingTenant124-456:789
        expr: 'tags.path:/var/log/httpd OR tags.path:/var/log/nginx | stats by (tags.host) count() requests'

```

--------------------------------

### OTLP Trace Span to VictoriaTraces Format Example

Source: https://docs.victoriametrics.com/victoriatraces/keyconcepts

This snippet demonstrates the transformation of an OpenTelemetry Protocol (OTLP) trace span into the VictoriaTraces format. It highlights how resource attributes and span details are mapped to VictoriaTraces fields, including '_stream', '_msg', and prefixed fields for resource and span attributes. The example assumes no specific programming language as it's a data format comparison.

```json
{
  "ResourceSpans": [{
    "Resource": {
      "Attributes": {
        "service.name": "payment",
        "telemetry.sdk.language": "nodejs",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.30.1",
        "container.id": "18ee03279d38ed0e0eedad037c260df78dfc3323aa662ca14a2d38fcc8bf3762",
        "service.namespace": "opentelemetry-demo",
        "service.version": "2.0.2",
        "host.name": "18ee03279d38",
        "host.arch": "arm64",
        "os.type": "linux",
        "os.version": "6.10.14-linuxkit",
        "process.pid": 17,
        "process.executable.name": "node",
        "process.executable.path": "/usr/local/bin/node",
        "process.command_args": [
          "/usr/local/bin/node",
          "--require",
          "./opentelemetry.js",
          "/usr/src/app/index.js"
        ],
        "process.runtime.version": "22.16.0",
        "process.runtime.name": "nodejs",
        "process.runtime.description": "Node.js",
        "process.command": "/usr/src/app/index.js",
        "process.owner": "node"
      }
    },
    "ScopeSpans": [{
      "Scope": {
        "Name": "@opentelemetry/instrumentation-net",
        "Version": "0.43.1",
        "Attributes": null,
        "DroppedAttributesCount": 0
      },
      "Spans": [{
        "TraceID": "769d28c4b8633dc9de2cc421d1a1616f",
        "SpanID": "2a1f3c4bda1d0e43",
        "TraceState": "",
        "ParentSpanID": "3ea61f2d2a5a003d",
        "Flags": 0,
        "Name": "tcp.connect",
        "Kind": 1,
        "StartTimeUnixNano": 1750044408780000000,
        "EndTimeUnixNano": 1750044408796828584,
        "Attributes": {
          "net.transport": "ip_tcp",
          "net.peer.name": "169.254.169.254",
          "net.peer.port": 80,
          "net.peer.ip": "169.254.169.254",
          "net.host.ip": "172.18.0.14",
          "net.host.port": 50722
        },
        "DroppedAttributesCount": 0,
        "Events": null,
        "DroppedEventsCount": 0,
        "Links": null,
        "DroppedLinksCount": 0,
        "Status": {
          "Message": "",
          "Code": 0
        }
      }],
      "SchemaURL": ""
    }],
    "SchemaURL": ""
  }]
}
```

```json
{
  "_time": "2025-06-16T03:26:48.796828584Z",
  "_stream_id": "00000000000000006fc12c07dedb6f693e73fc7b11b943e6",
  "_stream": "{name=\"tcp.connect\",resource_attr:service.name=\"payment\"}",
  "_msg": "-",
  "dropped_attributes_count": "0",
  "dropped_events_count": "0",
  "dropped_links_count": "0",
  "flags": "0",
  "kind": "1",
  "name": "tcp.connect",
  "resource_attr:container.id": "18ee03279d38ed0e0eedad037c260df78dfc3323aa662ca14a2d38fcc8bf3762",
  "resource_attr:host.arch": "arm64",
  "resource_attr:host.name": "18ee03279d38",
  "resource_attr:os.type": "linux",
  "resource_attr:os.version": "6.10.14-linuxkit",
  "resource_attr:process.command": "/usr/src/app/index.js",
  "resource_attr:process.command_args": "[\"/usr/local/bin/node\",\"--require\",\"./opentelemetry.js\",\"/usr/src/app/index.js\"]",
  "resource_attr:process.executable.name": "node",
  "resource_attr:process.executable.path": "/usr/local/bin/node",
  "resource_attr:process.owner": "node",
  "resource_attr:process.pid": "17",
  "resource_attr:process.runtime.description": "Node.js",
  "resource_attr:process.runtime.name": "nodejs",
  "resource_attr:process.runtime.version": "22.16.0",
  "resource_attr:service.name": "payment",
  "resource_attr:service.namespace": "opentelemetry-demo",
  "resource_attr:service.version": "2.0.2",
  "resource_attr:telemetry.sdk.language": "nodejs",
  "resource_attr:telemetry.sdk.name": "opentelemetry",
  "resource_attr:telemetry.sdk.version": "1.30.1",
  "scope_name": "@opentelemetry/instrumentation-net",
  "scope_version": "0.43.1",
  "span_attr:net.transport": "ip_tcp",
  "duration": "16828584",
  "end_time_unix_nano": "1750044408796828584",
  "parent_span_id": "3ea61f2d2a5a003d",
  "span_attr:net.host.ip": "172.18.0.14",
  "span_attr:net.host.port": "50722",
  "span_attr:net.peer.ip": "169.254.169.254",
  "span_attr:net.peer.name": "169.254.169.254",
  "span_attr:net.peer.port": "80",
  "span_id": "2a1f3c4bda1d0e43",
  "start_time_unix_nano": "1750044408780000000",
  "status_code": "0",
  "trace_id": "769d28c4b8633dc9de2cc421d1a1616f"
}
```

--------------------------------

### VictoriaTraces Alerting Rule Example (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

Example of an alerting rule group using LogsQL for VictoriaTraces. It defines an alert 'TooManySpans' that triggers if more than 50 spans are received for a specific span name within 5 minutes.

```yaml
groups:
  - name: CountBySpanName
    # The use of `type: vlogs` is because VictoriaTraces uses LogsQL (the same query language as VictoriaLogs).
    # No extra VictoriaLogs setup is needed during the process.
    type: vlogs
    interval: 5m
    rules:
       - alert: TooManySpans
         expr: '* | stats by (name) count() as total | filter total:>50'
         annotations:
            description: "Received more than 50 trace spans for span name {{$labels.name}} in the last 5 minute: {{$value}}"
    

```

--------------------------------

### Set vmalert Remote Write URL for Multitenant Persistence

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

This example shows how to configure the vmalert remote write URL to target the multitenant endpoint of vminsert. This allows different rule results to be persisted to different tenants in VictoriaMetrics based on configurations within the group.

```bash
./bin/vmalert -datasource.url=http://localhost:9428 -remoteWrite.url=http://vminsert:8480/insert/multitenant/prometheus ...
```

--------------------------------

### Combine Disk Limit with Extended Retention Period

Source: https://docs.victoriametrics.com/victoriatraces/index

Allows setting a disk space limit while ensuring trace spans are not dropped due to time-based retention by setting a very large `-retentionPeriod`. This example uses '10TiB' for disk space and '100y' for retention.

```bash
/path/to/victoria-traces -retention.maxDiskSpaceUsageBytes=10TiB -retentionPeriod=100y
```

```bash
/path/to/victoria-traces -retention.maxDiskUsagePercent=85 -retentionPeriod=100y
```

--------------------------------

### Specify Custom Storage Data Path for VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/index

Configures VictoriaTraces to store its data in a custom directory using the `-storageDataPath` flag. If the directory doesn't exist, VictoriaTraces creates it on the first run. This example sets the path to '/var/lib/victoria-traces'.

```bash
/path/to/victoria-traces -storageDataPath=/var/lib/victoria-traces
```

--------------------------------

### Enable VictoriaTraces OTLP gRPC Server

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry

Enables the OTLP gRPC server on VictoriaTraces to receive trace data. This involves starting the VictoriaTraces binary with specific flags for listening address and TLS configuration.

```bash
./victoria-traces -otlpGRPCListenAddr=:4317 -otlpGRPC.tlsCertFile=<cert_file> -otlpGRPC.tlsKeyFile=<key_file>
```

--------------------------------

### GET /select/jaeger/api/traces - Filter Traces by Tags

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This endpoint allows filtering traces based on span attributes, resource attributes, and instrumentation scope attributes using the `tags` query parameter.

```APIDOC
## GET /select/jaeger/api/traces

### Description
Filters traces by specifying attributes of spans, resources, or instrumentation scopes. The `tags` parameter accepts various formats for filtering.

### Method
GET

### Endpoint
`/select/jaeger/api/traces`

### Parameters
#### Query Parameters
- **tags** (string) - Required - A string containing attribute filters. 
  - **Span Attributes**: `<attribute_name>=<attribute_value>` (multiple can be space-separated).
  - **Resource Attributes**: `resource_attr:<attribute_name>=<attribute_value>` (multiple can be space-separated).
  - **Instrumentation Scope Attributes**: `scope_attr:<attribute_name>=<attribute_value>` (multiple can be space-separated).

### Request Example
```
GET /select/jaeger/api/traces?tags=error=unset%20otel.scope.name=redis-manual%20resource_attr:os.type=linux
```

### Response
#### Success Response (200)
Returns a list of traces that match the specified tag filters.

#### Response Example
(Response body structure depends on the traces returned and is not detailed here)

```

--------------------------------

### Query Data (Jaeger)

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

Access trace data through Jaeger Query Service JSON APIs for visualization in tools like Grafana.

```APIDOC
## GET /select/jaeger

### Description
This endpoint provides access to trace data via Jaeger Query Service JSON APIs. It is useful for integrating with systems like Grafana.

### Method
GET

### Endpoint
`/select/jaeger`

### Parameters
#### Query Parameters
- **service** (string) - Optional - Filter traces by service name.
- **operation** (string) - Optional - Filter traces by operation name.
- **limit** (integer) - Optional - The maximum number of traces to return.
- **start** (integer) - Optional - Start time for the query in milliseconds since epoch.
- **end** (integer) - Optional - End time for the query in milliseconds since epoch.

### Response
#### Success Response (200)
- **data** (object) - Contains trace data in Jaeger format.
  - **traces** (array) - An array of trace objects.

#### Response Example
```json
{
  "data": {
    "traces": [
      {
        "traceID": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
        "spans": [
          {
            "traceID": "a1b2c3d4e5f67890a1b2c3d4e5f67890",
            "spanID": "1234567890abcdef",
            "operationName": "http.request",
            "startTime": 1678886400000000000,
            "duration": 1000000,
            "tags": [
              {"key": "http.method", "type": "string", "value": "GET"},
              {"key": "http.url", "type": "string", "value": "/api/users"}
            ]
          }
        ]
      }
    ]
  }
}
```
```

--------------------------------

### Enable Basic Auth for vtinsert and vtselect clients

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtinsert and vtselect to use Basic Authentication when communicating with vtstorage. Set the `-storageNode.username` and `-storageNode.password` flags on the client components for secure authentication over untrusted networks.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls -storageNode.username=... -storageNode.password=...
```

--------------------------------

### Enable TLS for vtinsert and vtselect clients

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtinsert and vtselect to communicate with vtstorage over TLS by setting the `-storageNode.tls` command-line flag. This ensures encrypted data transfer when these components interact over untrusted networks.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls
```

--------------------------------

### Expose Internal Metrics in Prometheus Format

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

VictoriaTraces exposes its internal metrics in Prometheus exposition format. This endpoint can be scraped by Prometheus or other monitoring solutions to track the health and performance of VictoriaTraces itself.

```http
http://<victoria-traces>:10428/metrics
```

--------------------------------

### Enable TLS on vtstorage

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Configure vtstorage to accept incoming requests over HTTPS by specifying the `-tls`, `-tlsCertFile`, and `-tlsKeyFile` command-line flags. This is essential for secure communication over untrusted networks.

```bash
./victoria-traces-prod -httpListenAddr=... -storageDataPath=... -tls -tlsCertFile=/path/to/certfile -tlsKeyFile=/path/to/keyfile
```

--------------------------------

### Jaeger Query Service API Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

VictoriaTraces exposes trace data through Jaeger Query Service compatible APIs. This URL can be used to configure a Jaeger datasource in tools like Grafana for visualizing trace data.

```http
http://<victoria-traces>:10428/select/jaeger
```

--------------------------------

### Ingest Trace Data via OTLP/HTTP using curl

Source: https://docs.victoriametrics.com/victoriatraces/quick-start/index

This snippet demonstrates how to send a trace span to VictoriaTraces using a curl command. It constructs a JSON payload representing a trace span and sends it to the `/insert/opentelemetry/v1/traces` endpoint. The `ingestTimePlaceHolder` is dynamically replaced with the current time.

```shell
echo '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"frontend-web"}},{"key":"telemetry.sdk.language","value":{"stringValue":"webjs"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.30.1"}},{"key":"process.runtime.name","value":{"stringValue":"browser"}},{"key":"process.runtime.description","value":{"stringValue":"Web Browser"}},{"key":"process.runtime.version","value":{"stringValue":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.0.0 Safari/537.36"}}]},"scopeSpans":[{"scope":{"name":"@opentelemetry/instrumentation-document-load","version":"0.44.1"},"spans":[{"traceId":"1af5dd013a30efe7f2970032ab81958b","spanId":"229d083a6c480511","parentSpanId":"","name":"documentLoad","kind":1,"startTimeUnixNano":"ingestTimePlaceHolder","endTimeUnixNano":"ingestTimePlaceHolder","attributes":[{"key":"session.id","value":{"stringValue":"96e702c3-6f05-4f54-b2b3-2fad2b7b7995"}},{"key":"http.url","value":{"stringValue":"http://frontend-proxy:8080/cart"}},{"key":"http.user_agent","value":{"stringValue":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/136.0.0.0 Safari/537.36"}}],"events":[{"timeUnixNano":"1757320936519100098","name":"fetchStart"}],"status":{}}]}]}]}' |
sed "s/ingestTimePlaceHolder/$(date +%s000000000)/g" |
curl -X POST -H 'Content-Type: application/json' --data-binary @- http://<victoria-traces>:10428/insert/opentelemetry/v1/traces
```

--------------------------------

### Download and Unpack VictoriaTraces Executable

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Downloads the latest VictoriaTraces release for Linux amd64 and unpacks the archive. This is the initial step before setting up the cluster.

```bash
curl -L -O https://github.com/VictoriaMetrics/VictoriaTraces/releases/download/v0.6.0/victoria-traces-linux-amd64-v0.6.0.tar.gz
tar xzf victoria-traces-linux-amd64-v0.6.0.tar.gz
```

--------------------------------

### Run vmalert with VictoriaTraces Datasource

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to run vmalert, specifying the path to rule files, the VictoriaTraces datasource URL, the AlertManager URL, and remote write/read URLs for state persistence and restoration.

```bash
./bin/vmalert -rule=alert.rules                  \
    -datasource.url=http://victoriatraces:10428  \
    -notifier.url=http://alertmanager:9093       \
    -remoteWrite.url=http://victoriametrics:8428 \
    -remoteRead.url=http://victoriametrics:8428  \
    

```

--------------------------------

### vmalert Alerting Rule Example for VictoriaTraces (LogsQL)

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

An example of an alerting rule configuration for vmalert using LogsQL, suitable for VictoriaTraces. This defines a rule named 'TooManySpans' that triggers when more than 50 spans are received for a span name in the last 5 minutes.

```yaml
groups:
  - name: CountBySpanName
    type: vlogs
    interval: 5m
    rules:
       - alert: TooManySpans
         expr: '* | stats by (name) count() as total | filter total:>50'
         annotations:
            description: "Received more than 50 trace spans for span name {{$labels.name}} in the last 5 minute: {{$value}}"
```

--------------------------------

### Configure client TLS certificates for mTLS

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Specify client TLS certificates for vtinsert and vtselect nodes when using mTLS with vtstorage. Use the `-storageNode.tlsCertFile` and `-storageNode.tlsKeyFile` flags.

```bash
./victoria-traces-prod -storageNode=... -storageNode.tls -storageNode.tlsCertFile=/path/to/client.cert -storageNode.tlsKeyFile=/path/to/client.key
```

--------------------------------

### Run Enterprise vmalert with Cluster Mode for Multitenancy

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to run the enterprise version of vmalert with cluster mode enabled. This approach allows specifying the tenant directly per group configuration, facilitating multitenancy for VictoriaTraces.

```bash
./bin/vmalert -datasource.url=http://localhost:9428 -clusterMode=true -remoteWrite.url=http://vminsert:8480/ ...

```

--------------------------------

### vmalert Rule Backfilling Command

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command demonstrates how to perform rule backfilling (replay) against VictoriaTraces using vmalert. It specifies the path to the rules file, the VictoriaTraces datasource URL, the default rule type, a remote write URL for state persistence, and the time range for the replay.

```bash
./bin/vmalert -rule=path/to/your.rules \
    -datasource.url=http://localhost:10428 \
    -rule.defaultRuleType=vlogs \
    -remoteWrite.url=http://localhost:8428 \
    -replay.timeFrom=2021-05-11T07:21:43Z \
    -replay.timeTo=2021-05-29T18:40:43Z
```

--------------------------------

### Build vtgen Trace Data Generator

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Clones the VictoriaTraces repository and builds the vtgen tool, used for generating trace data. This can be done using 'go build' or Docker.

```bash
# clone the source code
git clone https://github.com/VictoriaMetrics/VictoriaTraces.git
cd VictoriaTraces

# build via go build
make vtgen
# build via Docker
make vtgen-prod
```

--------------------------------

### Enable Basic Auth for vtstorage

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Secure HTTPS requests to vtstorage by enabling Basic Authentication. Specify the `-httpAuth.username` and `-httpAuth.password` flags on vtstorage to verify credentials in incoming requests.

```bash
./victoria-traces-prod -httpListenAddr=... -storageDataPath=... -tls -tlsCertFile=... -tlsKeyFile=... -httpAuth.username=... -httpAuth.password=...
```

--------------------------------

### Run vmalert with Cluster Mode for Tenant Specification

Source: https://docs.victoriametrics.com/victoriatraces/vmalert

This command demonstrates how to run vmalert with cluster mode enabled, allowing tenant specification directly within the group configuration. This approach is used in conjunction with the enterprise version of vmalert.

```bash
./bin/vmalert -datasource.url=http://localhost:9428 -clusterMode=true -remoteWrite.url=http://vminsert:8480/ ...
```

--------------------------------

### List All Services

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

Retrieves a list of all available services within the tracing system.

```APIDOC
## GET /select/jaeger/api/services

### Description
Retrieves a list of all services instrumented for tracing.

### Method
GET

### Endpoint
/select/jaeger/api/services

### Parameters
#### Query Parameters
None

#### Request Body
None

### Response
#### Success Response (200)
- **data** (array) - A list of service names.
- **errors** (null) - Should be null if no errors occurred.
- **limit** (integer) - The limit applied to the query.
- **offset** (integer) - The offset applied to the query.
- **total** (integer) - The total number of services available.

#### Response Example
```json
{
    "data": [
        "accounting",
        "ad",
        "cart",
        "checkout",
        "currency"
    ],
    "errors": null,
    "limit": 0,
    "offset": 0,
    "total": 5
}
```
```

--------------------------------

### Build VictoriaTraces Docker Image (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Builds a Docker image for VictoriaTraces. The image tag is auto-generated but can be manually set. Allows specifying a custom base image.

```bash
make package-victoria-traces
# To set a custom tag:
PKG_TAG=foobar make package-victoria-traces
# To use a custom base image (e.g., scratch):
ROOT_IMAGE=scratch make package-victoria-traces
```

--------------------------------

### Find Trace by Trace ID using cURL

Source: https://docs.victoriametrics.com/victoriatraces/querying/index

This snippet demonstrates how to retrieve a specific trace from the Victoria Traces API using its unique trace ID. It utilizes the cURL command-line tool to send an HTTP GET request to the '/select/jaeger/api/traces/{trace_id}' endpoint.

```bash
curl http://<victoria-traces>:10428/select/jaeger/api/traces/9e06226196051d9c3c10dfab343791ad
```

--------------------------------

### Set Absolute Disk Space Limit for VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/index

Configures VictoriaTraces to drop old partitions when total data size exceeds a specified byte limit. The `-retention.maxDiskSpaceUsageBytes` flag sets this threshold. For example, setting it to '100GiB' limits the storage to that amount.

```bash
/path/to/victoria-traces -retention.maxDiskSpaceUsageBytes=100GiB
```

--------------------------------

### Enable mTLS on vtstorage

Source: https://docs.victoriametrics.com/victoriatraces/cluster

Enable mutual TLS (mTLS) on vtstorage to verify client TLS certificates from vtinsert and vtselect nodes. This requires the `-mtls` flag in addition to `-tls` on vtstorage.

```bash
# Example assuming TLS is already configured
./victoria-traces-prod -httpListenAddr=... -storageDataPath=... -tls -tlsCertFile=... -tlsKeyFile=... -mtls
```

--------------------------------

### Clone VictoriaTraces Repository (Bash)

Source: https://docs.victoriametrics.com/victoriatraces/index

Clones the VictoriaTraces Git repository to the local machine. This is the first step in building from sources.

```bash
git clone https://github.com/VictoriaMetrics/VictoriaTraces.git;
cd VictoriaTraces;
```

--------------------------------

### Configure vmalert with vmauth for Mixed Datasources

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command shows how to configure vmalert to use vmauth as its datasource URL. vmauth will then handle routing queries to either VictoriaMetrics or VictoriaTraces based on the configured rules.

```bash
./bin/vmalert -datasource.url=http://vmauth:8427/ ...

```

--------------------------------

### Set Percentage-Based Disk Space Limit for VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/index

Sets a dynamic disk space limit for VictoriaTraces based on the filesystem's total capacity using the `-retention.maxDiskUsagePercent` flag. This is useful for cloud environments or maintaining consistent free space. For example, '80%' limits usage to 80% of the filesystem.

```bash
/path/to/victoria-traces -retention.maxDiskUsagePercent=80
```

--------------------------------

### Configure Go SDK gRPC Exporter for VictoriaTraces

Source: https://docs.victoriametrics.com/victoriatraces/data-ingestion/opentelemetry

Sets up a gRPC exporter for the Go OpenTelemetry SDK to send trace data to VictoriaTraces. Requires specifying the `Endpoint` and optionally TLS credentials. An option to use an insecure connection is commented out.

```go
traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("<victoria-traces>:4317"),
    otlptracegrpc.WithTLSCredentials(<creds>),
    // otlptracegrpc.WithInsecure(),  // use insecure connection if TLS is disabled.
)
```

--------------------------------

### Run vmalert with Multitenant vminsert Endpoint

Source: https://docs.victoriametrics.com/victoriatraces/vmalert/index

This command shows how to configure vmalert to use the multitenant endpoint of vminsert for remote writing. This allows different rule groups to persist data to different tenants in VictoriaMetrics by specifying tenant labels within the group configuration.

```bash
./bin/vmalert -datasource.url=http://localhost:9428 -remoteWrite.url=http://vminsert:8480/insert/multitenant/prometheus ...

```