VictoriaLogs

https://docs.victoriametrics.com/victorialogs/
Admin
VictoriaLogs is an open-source, resource-efficient, and fast database for logs from VictoriaMetrics,...
Tokens:360,582
Snippets:2,909
Trust Score:10
Update:9 months ago
Show doc for...
Tokens:
Raw
### Get VictoriaLogs Help and Configuration Info

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Displays the help message for VictoriaLogs, listing all supported command-line flags, their descriptions, and default values. This is used to understand configuration options.

```bash
/path/to/victoria-logs -help
```

--------------------------------

### Run VictoriaLogs using Docker Image

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Starts VictoriaLogs in a Docker container, mapping the necessary port and volume for data storage. This is a simple way to begin using VictoriaLogs.

```bash
docker run --rm -it -p 9428:9428 -v ./victoria-logs-data:/victoria-logs-data \
docker.io/victoriametrics/victoria-logs:v1.43.1 -storageDataPath=victoria-logs-data
```

--------------------------------

### Build VictoriaLogs from Source (Requires Go)

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Builds the VictoriaLogs executable from source code using the 'make' command. This requires Go to be installed on the system.

```bash
make victoria-logs
```

--------------------------------

### Build VictoriaLogs using Docker Builder

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Builds the victoria-logs-prod executable using a Docker builder container, which does not require Go to be installed locally. Docker must be installed.

```bash
make victoria-logs-prod
```

--------------------------------

### Run VictoriaLogs with Pre-built Binaries (Linux/amd64)

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Downloads, unpacks, and runs the VictoriaLogs pre-built binary for Linux amd64. This method has no external dependencies and automatically adapts to system resources.

```bash
curl -L -O https://github.com/VictoriaMetrics/VictoriaLogs/releases/download/v1.43.1/victoria-logs-linux-amd64-v1.43.1.tar.gz
tar xzf victoria-logs-linux-amd64-v1.43.1.tar.gz
./victoria-logs-prod -storageDataPath=victoria-logs-data
```

--------------------------------

### Run VictoriaLogs from Built Binary

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Executes the VictoriaLogs binary that has been built from source. It specifies the data storage path.

```bash
bin/victoria-logs -storageDataPath=victoria-logs-data
```

--------------------------------

### Start vlstorage Node

Source: https://docs.victoriametrics.com/victorialogs/cluster

Starts a `vlstorage` node, which listens on a specified HTTP port and stores data in a given directory.

```APIDOC
## Start vlstorage Node

Starts a single `vlstorage` node. This node accepts incoming requests on the specified HTTP listen address and stores ingested logs in the provided storage data path. The process is started in the background.

### Method
N/A (Command Execution)

### Endpoint
N/A (Local Process)

### Parameters
#### Command-line Flags
- **-httpListenAddr** (string) - Required - The address and port for the node to listen on (e.g., `:9491`).
- **-storageDataPath** (string) - Required - The directory where the node will store its data (e.g., `victoria-logs-data-1`).

### Request Example
```bash
./victoria-logs-prod -httpListenAddr=:9491 -storageDataPath=victoria-logs-data-1 &
```

### Response
N/A (Background process started)
```

--------------------------------

### Start vlselect Node

Source: https://docs.victoriametrics.com/victorialogs/cluster

Starts a `vlselect` node that accepts query requests and retrieves data from `vlstorage` nodes.

```APIDOC
## Start vlselect Node

Starts a `vlselect` node. This node accepts incoming query requests on its HTTP listen address and fetches the necessary data from the specified `vlstorage` nodes. The process is started in the background.

### Method
N/A (Command Execution)

### Endpoint
N/A (Local Process)

### Parameters
#### Command-line Flags
- **-httpListenAddr** (string) - Required - The address and port for the `vlselect` node to listen on (e.g., `:9471`).
- **-storageNode** (string) - Required - A comma-separated list of `vlstorage` node addresses (e.g., `localhost:9491,localhost:9492`).

### Request Example
```bash
./victoria-logs-prod -httpListenAddr=:9471 -storageNode=localhost:9491,localhost:9492 &
```

### Response
N/A (Background process started)
```

--------------------------------

### Clone VictoriaLogs Repository

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Clones the VictoriaLogs source code repository from GitHub. This is the first step for building VictoriaLogs from source.

```bash
git clone https://github.com/VictoriaMetrics/VictoriaLogs
cd VictoriaLogs
```

--------------------------------

### Start VictoriaLogs Storage Node

Source: https://docs.victoriametrics.com/victorialogs/cluster/index

This command starts a `vlstorage` node for VictoriaLogs. It listens on port 9491 for incoming requests and stores data in the 'victoria-logs-data-1' directory. The '&' runs the process in the background.

```bash
./victoria-logs-prod -httpListenAddr=:9491 -storageDataPath=victoria-logs-data-1 &
```

--------------------------------

### Start vlinsert Node

Source: https://docs.victoriametrics.com/victorialogs/cluster

Starts a `vlinsert` node responsible for accepting logs and distributing them across `vlstorage` nodes.

```APIDOC
## Start vlinsert Node

Starts a `vlinsert` node. This node accepts incoming logs on its HTTP listen address and distributes them evenly across the specified `vlstorage` nodes. The process is started in the background.

### Method
N/A (Command Execution)

### Endpoint
N/A (Local Process)

### Parameters
#### Command-line Flags
- **-httpListenAddr** (string) - Required - The address and port for the `vlinsert` node to listen on (e.g., `:9481`).
- **-storageNode** (string) - Required - A comma-separated list of `vlstorage` node addresses (e.g., `localhost:9491,localhost:9492`).

### Request Example
```bash
./victoria-logs-prod -httpListenAddr=:9481 -storageNode=localhost:9491,localhost:9492 &
```

### Response
N/A (Background process started)
```

--------------------------------

### Start VictoriaLogs Insert Node

Source: https://docs.victoriametrics.com/victorialogs/cluster/index

This command starts the `vlinsert` node, responsible for receiving logs and distributing them to the configured storage nodes. It listens on port 9481 and directs data to the two `vlstorage` nodes.

```bash
./victoria-logs-prod -httpListenAddr=:9481 -storageNode=localhost:9491,localhost:9492 &
```

--------------------------------

### Build vlogscli-prod with Make and Docker

Source: https://docs.victoriametrics.com/victorialogs/querying/vlogscli/index

Builds the `vlogscli-prod` binary using the Make utility within a Docker container. This method does not require a local Go installation but needs Docker to be installed.

```bash
make vlogscli-prod
```

--------------------------------

### SQL for Top N Values vs. LogsQL Top Pipe Shortcut

Source: https://docs.victoriametrics.com/victorialogs/sql-to-logsql/index

Demonstrates the conversion of a common SQL pattern for finding top N values using GROUP BY, ORDER BY, and LIMIT into the concise LogsQL 'top' pipe shortcut.

```SQL
SELECT user_id, count(*) hits FROM table GROUP BY user_id ORDER BY hits DESC LIMIT 5
```

--------------------------------

### Build vlagent Locally

Source: https://docs.victoriametrics.com/victorialogs/vlagent

Builds the vlagent binary from the source code. This requires Go to be installed on the system. The resulting binary will be placed in the 'bin' directory.

```makefile
make vlagent
```

--------------------------------

### Start VictoriaLogs with TCP Syslog Listener on Port 514

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog

Starts VictoriaLogs, enabling it to accept syslog formatted logs on TCP port 514 across all network interfaces. This is a basic setup for receiving syslog data via TCP.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514
```

--------------------------------

### Start VictoriaLogs with Syslog TCP Listen Address

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

This command starts VictoriaLogs and configures it to listen for Syslog messages on TCP port 514 on all available network interfaces. It supports RFC3164 and RFC5424 formats.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514

```

--------------------------------

### Query Streams with Tenant Headers using cURL

Source: https://docs.victoriametrics.com/victorialogs/querying

This example shows how to query for streams while specifying a particular tenant using AccountID and ProjectID HTTP headers. The query targets logs from the last 5 minutes for tenant (AccountID=12, ProjectID=34).

```bash
curl http://localhost:9428/select/logsql/streams -H 'AccountID: 12' -H 'ProjectID: 34' -d 'query=_time:5m'
```

--------------------------------

### Calculate Stats by Time Buckets

Source: https://docs.victoriametrics.com/victorialogs/logsql/index

This example calculates the number of logs and unique IP addresses per minute (`_time:1m`) over the last 5 minutes, aliased as `logs_total` and `ips_total`.

```LogsQL
      _time:5m | stats by (_time:1m) count() logs_total, count_uniq(ip) ips_total
    
```

--------------------------------

### Start VictoriaLogs with Syslog TCP and UDP Listen Addresses

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

This command starts VictoriaLogs, enabling it to receive Syslog data simultaneously via TCP and UDP on port 514. This allows for flexible log ingestion from different sources.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.listenAddr.udp=:514

```

--------------------------------

### Start VictoriaLogs with TCP Syslog Listener and Custom Timezone

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog

Starts VictoriaLogs with a TCP syslog listener and sets the timezone for parsing RFC3164 timestamps to 'Europe/Berlin'. This ensures accurate timestamp interpretation when logs originate from different time zones.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.timezone='Europe/Berlin'
```

--------------------------------

### LogsQL Comments Example

Source: https://docs.victoriametrics.com/victorialogs/logsql/index

Demonstrates how to use comments in LogsQL queries. Comments start with '#' and extend to the end of the line, aiding in query readability and explanation.

```logsQL
      error                               # find logs with `error` word
  | stats by (_stream) count() logs # then count the number of logs per `_stream` label
  | sort by (logs) desc             # then sort by the found logs in descending order
  | limit 5                         # and show top 5 streams with the biggest number of logs
    
```

--------------------------------

### Query Tenant IDs with cURL

Source: https://docs.victoriametrics.com/victorialogs/querying/index

This example demonstrates how to query for tenant IDs within a specified time range using `curl`. It highlights the requirement to call this endpoint with an empty `AccountID` header for security.

```bash
curl http://localhost:9428/select/tenant_ids?start=<start>&end=<end>

```

--------------------------------

### Configure Docker for Multi-Platform Builds

Source: https://docs.victoriametrics.com/victorialogs/release-guide/index

Sets up Docker Buildx for multi-platform image building, specifically using QEMU for emulation. This is a prerequisite for building Docker images on various architectures.

```bash
docker buildx create --use --name=qemu
docker buildx inspect --bootstrap

```

--------------------------------

### DataDog Agent Log Ingestion Configuration

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/index

Enables log ingestion for the DataDog agent and specifies the VictoriaLogs endpoint (VMAuth) to send collected logs. This setup requires enabling logs and setting the custom URL for log data transmission.

```yaml
logs_enabled: true
logs_config:
  logs_dd_url: "<vmauth-base-url>"
  use_http: true
    

```

--------------------------------

### Filter Kubernetes Logs with vlagent

Source: https://docs.victoriametrics.com/victorialogs/vlagent

Example of starting vlagent with a filter to exclude logs based on Kubernetes pod annotations and namespaces. This demonstrates the use of the -kubernetesCollector.excludeFilter flag.

```bash
./vlagent -remoteWrite.url=http://victoria-logs:9428/insert/native -kubernetesCollector \
  -kubernetesCollector.excludeFilter='kubernetes.pod_annotation.logging.vlagent.io/exclude:=true or kubernetes.pod_namespace:in(test, logging)'
    
```

--------------------------------

### Loki Push API Endpoint

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/promtail

The primary endpoint for pushing logs to VictoriaLogs using a Loki-compatible client. This example shows a basic configuration.

```APIDOC
## POST /insert/loki/api/v1/push

### Description
This endpoint receives log data in the Loki push format from compatible log shippers.

### Method
POST

### Endpoint
`/insert/loki/api/v1/push`

### Parameters
#### Query Parameters
- **disable_message_parsing** (boolean) - Optional - If set to `1`, disables automatic JSON parsing of log messages. Default is `false`.
- **_stream_fields** (string) - Optional - A comma-separated list of labels to be treated as stream fields. Other labels are stored as log fields.
- **debug** (boolean) - Optional - If set to `1`, enables debug logging in VictoriaLogs for ingestion. Default is `false`.
- **ignore_fields** (string) - Optional - A comma-separated list of fields to ignore during ingestion.

#### Request Body
This endpoint expects a standard Loki push request body, typically containing streams of log lines with labels and timestamps.

### Request Example
```json
{
  "streams": [
    {
      "stream": {
        "job": "myjob",
        "instance": "myinstance"
      },
      "values": [
        [
          "1678886400000000000",
          "log line 1"
        ],
        [
          "1678886460000000000",
          "log line 2 with {\"key\": \"value\"}"
        ]
      ]
    }
  ]
}
```

### Response
#### Success Response (200)
An empty response with a 200 OK status indicates successful ingestion.

#### Response Example
(No content on success)
```

--------------------------------

### Checkout Specific Commit

Source: https://docs.victoriametrics.com/victorialogs/quickstart/index

Checks out a specific commit hash within the VictoriaLogs repository. This is useful for working with a particular version of the source code.

```bash
git checkout <commit-hash-here>
```

--------------------------------

### Query Log Hits Stats with Tenant Headers

Source: https://docs.victoriametrics.com/victorialogs/querying

This example demonstrates how to query log hits statistics for a specific tenant by providing `AccountID` and `ProjectID` in the HTTP headers. This is essential when working with multi-tenant VictoriaLogs deployments. If these headers are omitted, the default tenant is queried.

```Shell
curl http://localhost:9428/select/logsql/hits -H 'AccountID: 12' -H 'ProjectID: 34' -d 'query=error'
```

--------------------------------

### Query Stream Field Names with cURL

Source: https://docs.victoriametrics.com/victorialogs/querying/index

This example demonstrates how to use `curl` to query for stream field names within a specified time range and query. It shows basic usage and how to specify tenant IDs via headers.

```bash
curl http://localhost:9428/select/logsql/stream_field_names -d 'query=error' -d 'start=5m'

```

```bash
curl http://localhost:9428/select/logsql/stream_field_names -H 'AccountID: 12' -H 'ProjectID: 34' -d 'query=_time:5m'

```

--------------------------------

### Configure Compression for Syslog Ingestion

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

Enable compression for syslog messages ingested by VictoriaLogs. Supported methods include 'none', 'zstd', 'gzip', and 'deflate'. This example configures gzip compression for TCP.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.compressMethod.tcp=gzip
```

--------------------------------

### LogsQL Top Pipe Shortcut for Top N Values

Source: https://docs.victoriametrics.com/victorialogs/sql-to-logsql/index

Provides the equivalent LogsQL query using the 'top' pipe shortcut to achieve the same result as the SQL query for finding top N values.

```LogsQL
* | top 5 (user_id)
```

--------------------------------

### Start VictoriaLogs Select Node

Source: https://docs.victoriametrics.com/victorialogs/cluster/index

This command initiates the `vlselect` node, which handles incoming queries and retrieves data from the storage nodes. It listens on port 9471 and fetches data from the specified `vlstorage` nodes.

```bash
./victoria-logs-prod -httpListenAddr=:9471 -storageNode=localhost:9491,localhost:9492 &
```

--------------------------------

### Get Unique Field Values using field_values Pipe

Source: https://docs.victoriametrics.com/victorialogs/logsql-examples/index

Retrieves all unique values for a given field (e.g., 'level') over the last 5 minutes using the 'field_values' pipe. This is an alternative to 'stats by (field)'.

```LogsQL
_time:5m | field_values level
```

--------------------------------

### Sort Log Results with External Tools

Source: https://docs.victoriametrics.com/victorialogs/querying

This example shows how to retrieve all log entries matching a query and sort them using the `sort` command after piping the output from `curl`. Be aware that this can be time-consuming if the query returns a large number of results.

```shell
curl http://localhost:9428/select/logsql/query -d 'query=error | sort by (_time)' | less
```

--------------------------------

### Configure Custom Stream Fields for Syslog Ingestion

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

Define custom fields to be used as log stream identifiers. This example sets 'hostname' and 'app_name' as stream fields for logs received on TCP port 514.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.streamFields.tcp='["hostname","app_name"]'
```

--------------------------------

### Configure Multitenancy for Syslog Ingestion

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

Specify a custom tenant for ingesting syslog messages. This example configures VictoriaLogs to store messages received on TCP port 514 into the tenant with AccountID 12 and ProjectID 34.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.tenantID.tcp=12:34
```

--------------------------------

### Count Log Entries with wc -l

Source: https://docs.victoriametrics.com/victorialogs/querying

This example demonstrates counting the number of log messages matching specific criteria using the `wc -l` Unix command. It queries for logs with the 'error' word and the field `app="nginx"` within the last 5 minutes.

```bash
curl http://localhost:9428/select/logsql/query -d 'query={app="nginx"} AND _time:5m AND error' | wc -l
```

--------------------------------

### Get Unique Sets of Field Values using uniq Pipe

Source: https://docs.victoriametrics.com/victorialogs/logsql-examples/index

Returns unique combinations of values for multiple specified fields (e.g., 'host', 'path') over the last 5 minutes using the 'uniq' pipe.

```LogsQL
_time:5m | uniq by (host, path)
```

--------------------------------

### Build vlagent from Source with Make

Source: https://docs.victoriametrics.com/victorialogs/vlagent/index

These commands demonstrate how to build the vlagent binary from the VictoriaLogs source code. It involves cloning the repository, checking out a specific commit if necessary, and then using the 'make' command to compile. Requires Go to be installed.

```shell
git clone https://github.com/VictoriaMetrics/VictoriaLogs
cd VictoriaLogs
git checkout <commit-hash-here>
make vlagent
```

--------------------------------

### Get Unique Field Values using uniq Pipe

Source: https://docs.victoriametrics.com/victorialogs/logsql-examples/index

Returns unique values for a specified field (e.g., 'ip') over the last 5 minutes using the 'uniq' pipe. This is useful for identifying distinct entries.

```LogsQL
_time:5m | uniq by (ip)
```

--------------------------------

### Limit Query Results with head

Source: https://docs.victoriametrics.com/victorialogs/querying

This example shows how to limit the initial investigation of query results to the first 10 log entries using the `head` command. Piping to `head` causes the query to be automatically canceled once 10 entries are retrieved, conserving resources.

```bash
curl http://localhost:9428/select/logsql/query -d 'query=error' | head -10
```

--------------------------------

### Configure Dropping Fields for Syslog Ingestion

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

Specify fields to be ignored during syslog ingestion. This example drops 'prod_id' and 'msg_id' fields for logs received on TCP port 514. Wildcards can be used to ignore field prefixes.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.ignoreFields.tcp='["prod_id","msg_id"]'
```

--------------------------------

### Serverless DataDog Plugin VictoriaLogs Endpoint Configuration

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/datadog-agent/index

Sets the VictoriaLogs endpoint for the Serverless DataDog plugin using the LOGS_DD_URL environment variable. This ensures logs are sent to the correct VMAuth proxy address.

```yaml
custom:
  datadog:
    apiKey: "fakekey"                 # Set any key, otherwise plugin fails
provider:
  environment:
    DD_DD_URL: "<vmauth-base-url>/"   # VMAuth endpoint for DataDog
    

```

--------------------------------

### Query Stream Field Values with cURL

Source: https://docs.victoriametrics.com/victorialogs/querying/index

This example shows how to use `curl` to query for specific field values within logs, given a query and time range. It includes an example with a `limit` parameter and how to set tenant IDs.

```bash
curl http://localhost:9428/select/logsql/stream_field_values -d 'query=error' -d 'start=5m' -d 'field=host'

```

```bash
curl http://localhost:9428/select/logsql/stream_field_values -H 'AccountID: 12' -H 'ProjectID: 34' -d 'query=_time:5m'

```

--------------------------------

### Configure Syslog Timezone in VictoriaLogs

Source: https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/index

This configuration starts VictoriaLogs to accept Syslog data on TCP port 514 and specifies 'Europe/Berlin' as the timezone for parsing RFC3164 timestamps. This ensures accurate time logging across different regions.

```bash
./victoria-logs -syslog.listenAddr.tcp=:514 -syslog.timezone='Europe/Berlin'

```

--------------------------------

### Build vlagent using Docker

Source: https://docs.victoriametrics.com/victorialogs/vlagent

Builds the vlagent binary using a Docker builder container. This method does not require Go to be installed locally but requires Docker. The resulting executable is named 'victoria-logs-prod' and placed in the 'bin' folder.

```makefile
make vlagent-prod
```

--------------------------------

### Detecting Most Frequent Logs with collapse_nums Pipe

Source: https://docs.victoriametrics.com/victorialogs/faq/index

Use the `collapse_nums` pipe to identify the most frequently occurring log messages over a specified time range. This example shows how to get the top 10 most frequent messages in the last hour, optionally including the stream field.

```LogsQL
_time:1h | collapse_nums prettify | top 10 (_msg)

```

```LogsQL
_time:1h | collapse_nums prettify | top 10 (_stream, _msg)

```
Tokens:
Raw