Spl

https://github.com/xingyunyang01/spl
Admin
Project description could not be determined as no content was provided for analysis.
Tokens:4,085
Snippets:32
Trust Score:6.6
Update:1 year ago
Show doc for...
Tokens:
Raw
### SPL Scan Query Example

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

An example of a scan query in SPL that first filters logs based on 'error' and 'msg:wrong', and then projects the 'level' and 'msg' fields. This demonstrates filtering and field selection capabilities.

```spl
error and msg:wrong | project level, msg
```

--------------------------------

### Access Array Elements in SPL

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

Demonstrates how to access elements within an array in SPL. Array indexing starts from 1. Example shows creating an array and accessing its second element.

```SPL
* | extend a = ARRAY[0, 1, 2] | extend b = a[1]
```

--------------------------------

### SPL Wildcard Input for Real-time Consumption

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

Shows how to use the wildcard '*' to input all raw data from a Logstore for real-time consumption using SPL. This enables filtering and analysis of all incoming logs, with field names being case-sensitive. An example filters logs containing 'wrong'.

```spl
* | where msg like '%wrong%'
```

--------------------------------

### SPL: Using '*' as a Placeholder for Input Data

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

The '*' symbol in SPL serves as a placeholder for SLS Logstore data when it's used as input. This example demonstrates filtering and categorizing access logs based on status codes using SPL.

```sql
-- SPL processing result defined as a named dataset src, as input for subsequent SPL expressions.
.let src = *
| where status=cast(status as BIGINT);
-- Using the named dataset src as input, filter data where the status field is 5xx, define the dataset err, do not output.
.let err = $src
| where status >= 500
| extend msg='ERR';

-- Using the named dataset src as input, filter data where the status field is 2xx, define the dataset ok, do not output.
.let ok = $src
| where status >= 200 and status < 300
| extend msg='OK';

-- Output named datasets err and ok
$err; $ok;
```

--------------------------------

### Access Map Elements in SPL

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

Illustrates how to access elements within a map (dictionary) in SPL. Map keys must be primitive numerical types. Example shows creating a map and accessing an element using a key.

```SPL
* | extend a = MAP(ARRAY['foo', 'bar'], ARRAY[0, 1]) | extend b = a['foo']
```

--------------------------------

### SPL Named Dataset Definition with .let

Source: https://context7.com/xingyunyang01/spl/llms.txt

Create reusable named datasets for complex multi-branch processing workflows using the '.let' control instruction. This enables modularity and reusability of data processing logic.

```sql
-- Define base dataset with type conversion
.let src = *
  | where status=cast(status as BIGINT);

-- Branch 1: Process error logs
.let err = $src
  | where status >= 500
  | extend msg='ERR', severity=3;

-- Branch 2: Process success logs
.let ok = $src
  | where status >= 200 and status < 300
  | extend msg='OK', severity=0;

-- Output both datasets
$err;
$ok;

-- Input example:
-- status: '200', body: 'this is a test'
-- status: '500', body: 'internal error'
-- status: '404', body: 'not found'

-- Output:
-- Dataset err: status: '500', body: 'internal error', msg: 'ERR', severity: 3
-- Dataset ok: status: '200', body: 'this is a test', msg: 'OK', severity: 0
```

--------------------------------

### SPL Command Expression Syntax

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

This describes the general syntax for SPL command expressions. It shows how commands are structured with options, expressions, and output aliasing. This syntax is fundamental for writing any SPL query.

```spl
cmd -option=<option> -option ... <expression>, ... as <output>, ...
```

--------------------------------

### Basic SPL Query Pipeline Execution

Source: https://context7.com/xingyunyang01/spl/llms.txt

Execute a multi-stage log processing query using pipe operators to connect commands and filter results by field values. This pipeline allows for filtering, projecting, and limiting log data.

```sql
SELECT * FROM logstore
WHERE cast(status as BIGINT) >= 500
| project timestamp, level, err_msg, request_id
| limit 100;
```

--------------------------------

### Convert Logs to Metrics with SPL 'log-to-metric'

Source: https://context7.com/xingyunyang01/spl/llms.txt

Transform log format data into time-series metric format for metric storage. Allows specifying time fields, labels, and metrics.

```sql
-- Convert logs to metrics with labels
* | log-to-metric
    -time=timestamp
    -labels=host,service,region
    -metrics=request_count,error_rate,response_time;

-- Advanced metric conversion
* | extend metric_time = unix_timestamp()
  | log-to-metric
    -time=metric_time
    -labels=environment,app_name,instance_id
    -metrics=cpu_usage,memory_usage,disk_io;
```

--------------------------------

### SQL 表达式示例 - 数据计算

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

SQL 表达式用于数据计算，返回值可以是布尔值或计算结果。

```sql
cast(status as int)>=500
```

--------------------------------

### SPL Field Projection and Renaming with project

Source: https://context7.com/xingyunyang01/spl/llms.txt

Select and rename specific fields while discarding others using the 'project' instruction for schema transformation. Supports explicit field selection and wildcard matching.

```sql
-- Keep specific fields and rename them
* | project log_level=level, error_message=err_msg, timestamp;

-- Use wildcard mode to match field patterns
* | project -wildcard "__tag__:*", "user_*", status;

-- Input: level: 'ERROR', err_msg: 'Connection timeout', timestamp: '2025-10-16', user_id: '12345'
-- Output: log_level: 'ERROR', error_message: 'Connection timeout', timestamp: '2025-10-16'
```

--------------------------------

### SPL JSON Array Expansion with expand-values

Source: https://context7.com/xingyunyang01/spl/llms.txt

Expand JSON arrays or objects into multiple log entries, generating one row per element. Supports expanding specific JSON paths and retaining original fields.

```sql
-- Expand JSON array to multiple rows
* | expand-values items;

-- Input: order_id: '12345', items: '[{"name":"apple","qty":2},{"name":"banana","qty":5}]'
-- Output (2 entries):
-- Entry 1: order_id: '12345', items: '{"name":"apple","qty":2}'
-- Entry 2: order_id: '12345', items: '{"name":"banana","qty":5}'

-- Expand nested JSON path and keep original field
* | expand-values -path='$.body' -keep content as body;

-- Input: content: '{"body": [0, {"a": 1, "b": 2}]}'
-- Output (2 entries):
-- Entry 1: content: '{"body": [0, {"a": 1, "b": 2}]}', body: '0'
-- Entry 2: content: '{"body": [0, {"a": 1, "b": 2}]}', body: '{"a": 1, "b": 2}'
```

--------------------------------

### Pack Fields into JSON with SPL 'pack-fields'

Source: https://context7.com/xingyunyang01/spl/llms.txt

Pack multiple fields into a single JSON-serialized field for structured data transmission. Supports specific fields or wildcard patterns.

```sql
-- Pack specific fields into JSON object
* | pack-fields -output=request_data client_ip, method, path, query_string;

-- Pack with field pattern
* | pack-fields -wildcard -output=user_info "user_*";

-- Input: client_ip: '192.168.1.1', method: 'POST', path: '/api/login', status: 200
-- Output: request_data: '{"client_ip":"192.168.1.1","method":"POST","path":"/api/login"}', status: 200
```

--------------------------------

### Limit Query Results with SPL 'limit' Command

Source: https://context7.com/xingyunyang01/spl/llms.txt

Limit the number of output rows returned by the query. Useful for pagination and retrieving top results.

```sql
-- Get top 100 results
* | where status >= 400
  | sort timestamp desc
  | limit 100;

-- Pagination pattern
* | stats count() as error_count by error_type
  | sort error_count desc
  | limit 20;

-- Combined with aggregation
* | stats sum(bytes_sent) as total_bytes by client_ip
  | sort total_bytes desc
  | limit 5;
```

--------------------------------

### SPL 表达式示例 - 字段赋值

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

SPL 表达式支持直接对字段赋值，或使用正则表达式提取结果赋值给字段。

```spl
level='INFO'
```

```spl
msg=regex_extract(content, '\S+')
```

--------------------------------

### Extend Fields with SQL Expressions

Source: https://context7.com/xingyunyang01/spl/llms.txt

Create new fields by applying SQL expressions and functions to existing fields. Supports various data types and complex calculations.

```SQL
-- Add computed fields using SQL functions
* | extend
    duration_ms = cast(response_time as DOUBLE) * 1000,
    is_error = status >= 400,
    log_date = date_format(from_unixtime(timestamp), '%Y-%m-%d'),
    upper_method = upper(method);

-- Complex calculations with multiple functions
* | extend
    full_name = concat(first_name, ' ', last_name),
    age_years = date_diff('year', birth_date, current_date),
    status_category = case
        when status < 300 then 'success'
        when status < 400 then 'redirect'
        when status < 500 then 'client_error'
        else 'server_error'
    end;

-- Input: response_time: '0.25', status: 500, method: 'get'
-- Output: response_time: '0.25', status: 500, method: 'get', duration_ms: 250.0, is_error: true, upper_method: 'GET'
```

--------------------------------

### Transform Metrics with SPL 'metric-to-metric'

Source: https://context7.com/xingyunyang01/spl/llms.txt

Transform existing metric data by adding, modifying, or deleting labels and metric values. Supports adding, dropping, and renaming labels, and modifying metric calculations.

```sql
-- Add new labels to metrics
* | metric-to-metric
    -add_labels=cluster='prod',datacenter='us-east-1'
    -rename_labels=old_name:new_name;

-- Filter and transform metric labels
* | metric-to-metric
    -drop_labels=internal_id,debug_flag
    -modify_metric=request_count*1000;
```

--------------------------------

### SPL Wildcard Input for Logtail Collection

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

Demonstrates using the wildcard '*' to input all raw data collected by Logtail into an SPL query for parsing. This method is suitable when you need to process the entire raw log content. It highlights that field names are case-sensitive in this context.

```spl
* | parse-json content
```

--------------------------------

### SPL Wildcard Input for Ingest Processors

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

Explains the usage of the wildcard '*' to represent all raw data written by ingest processors as input for SPL queries. This is useful for processing data immediately after ingestion. Field names are case-sensitive.

```spl
*
```

--------------------------------

### SPL 表达式示例 - 字段名和通配符

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

SPL 表达式可以用于匹配字段名或使用通配符。如果字段名包含特殊字符，需要用双引号包裹。

```spl
content
```

```spl
"__tag__:__path__"
```

--------------------------------

### Perform Statistical Aggregations with GROUP BY

Source: https://context7.com/xingyunyang01/spl/llms.txt

Perform GROUP BY aggregations using SQL aggregate functions like COUNT, SUM, AVG, MAX, MIN. Supports calculating percentiles and conditional counts.

```SQL
-- Basic aggregation with grouping
* | stats
    count() as total_requests,
    sum(bytes_sent) as total_bytes,
    avg(response_time) as avg_response_time,
    max(response_time) as max_response_time
  by status, method;

-- Multiple aggregations with percentiles
* | stats
    count() as req_count,
    approx_percentile(response_time, 0.95) as p95_latency,
    approx_percentile(response_time, 0.99) as p99_latency,
    count_if(status >= 500) as error_count,
    count_if(status = 200) as success_count
  by date_trunc('hour', from_unixtime(timestamp)) as hour,
     method;

-- Output example:
-- status: 200, method: 'GET', total_requests: 15420, total_bytes: 45123456, avg_response_time: 0.123
-- status: 500, method: 'POST', total_requests: 38, total_bytes: 12456, avg_response_time: 2.456
```

--------------------------------

### Filter Data with SQL Boolean Expressions

Source: https://context7.com/xingyunyang01/spl/llms.txt

Filter log entries using SQL boolean expressions with comprehensive function support. Allows for complex filtering logic based on multiple conditions.

```SQL
-- Filter by multiple conditions
* | where status >= 500
    and method in ('POST', 'PUT', 'DELETE')
    and response_time > 1.0
    and client_ip like '192.168.%';

-- Filter with regex and date functions
* | where regexp_like(message, 'ERROR|FATAL')
    and date_diff('hour', from_unixtime(timestamp), current_timestamp()) < 24
    and user_agent not like '%bot%';

-- Complex boolean logic with SQL functions
* | where (status = 200 and response_time > 3.0)
    or (status >= 500)
    or (length(error_message) > 0 and position('timeout', lower(error_message)) > 0)
```

--------------------------------

### Parse Key-Value Pairs

Source: https://context7.com/xingyunyang01/spl/llms.txt

Extract key-value pairs from structured text using customizable delimiters. This is useful for parsing log messages or query strings.

```SQL
-- Parse standard key-value format
* | parse-kv -pair_delim=' ' -kv_delim='=' message;

-- Parse URL query string
* | parse-kv -pair_delim='&' -kv_delim='=' query_string;

-- Input: message: 'user=john status=active role=admin level=5'
-- Output: user: 'john', status: 'active', role: 'admin', level: '5'
```

--------------------------------

### Sort Query Results with SPL 'sort' Command

Source: https://context7.com/xingyunyang01/spl/llms.txt

Sort query results by one or more fields in ascending or descending order. Supports handling of null values.

```sql
-- Sort by single field descending
* | stats count() as req_count by client_ip
  | sort req_count desc
  | limit 10;

-- Multi-field sorting
* | stats
    count() as total,
    avg(response_time) as avg_time
  by status, method
  | sort status asc, avg_time desc;

-- Sort with nulls handling
* | sort timestamp desc nulls last, status asc;
```

--------------------------------

### SPL Field Removal with project-away

Source: https://context7.com/xingyunyang01/spl/llms.txt

Remove unwanted fields from the output while preserving all other fields using 'project-away'. This command is useful for data sanitization and security.

```sql
-- Remove sensitive fields from logs
* | project-away -wildcard "password*", "token*", "secret*";

-- Remove multiple specific fields
* | project-away internal_id, debug_info, temp_data;

-- Input: user: 'john', password: 'secret123', email: 'john@example.com', token: 'abc123'
-- Output: user: 'john', email: 'john@example.com'
```

--------------------------------

### Select and Rename Fields with 'project' in SPL

Source: https://github.com/xingyunyang01/spl/blob/main/字段操作指令.md

The 'project' command in SPL is used to select specific fields, optionally rename them, and control which fields are retained in the output. It supports wildcard matching for field patterns and allows renaming fields to new output names. If a new field name conflicts with an existing one, refer to the overwrite policy.

```spl
* | project level, err_msg
```

```spl
* | project log_level=level, err_msg
```

```spl
* | project "__tag__: *"
```

--------------------------------

### Parse Log Data with Regular Expressions

Source: https://context7.com/xingyunyang01/spl/llms.txt

Extract structured data from unstructured text fields using regular expression groups. This is useful for parsing various log formats into named fields for further analysis.

```SQL
-- Parse Apache log format
* | parse-regexp content '(?P<ip>[\d.]+) - (?P<user>\S+) \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+)"'
  | project ip, user, method, path;

-- Parse custom log format with named groups
* | parse-regexp message '^(?P<level>\w+):(?P<module>\w+):(?P<msg>.*)$';

-- Input: content: '192.168.1.1 - john [16/Oct/2025:10:30:00] "GET /api/users"'
-- Output: ip: '192.168.1.1', user: 'john', method: 'GET', path: '/api/users'
```

--------------------------------

### SPL Field Renaming with project-rename

Source: https://context7.com/xingyunyang01/spl/llms.txt

Rename fields without affecting other fields in the dataset using 'project-rename'. This is useful for standardizing field names across different data sources.

```sql
-- Rename multiple fields for standardization
* | project-rename
    request_time=req_time,
    response_time=resp_time,
    client_ip=remote_addr;

-- Input: req_time: '100ms', resp_time: '250ms', remote_addr: '192.168.1.1', status: '200'
-- Output: request_time: '100ms', response_time: '250ms', client_ip: '192.168.1.1', status: '200'
```

--------------------------------

### SPL 示例 - parse-csv 输出

Source: https://github.com/xingyunyang01/spl/blob/main/SPL基础语法.md

使用 parse-csv 命令从处理结果中输出指定字段。

```spl
| parse-csv content as a, b, c
```

--------------------------------

### Parse JSON Data

Source: https://context7.com/xingyunyang01/spl/llms.txt

Extract first-level JSON object properties into separate log fields. Optionally, the original JSON field can be preserved.

```SQL
-- Parse JSON content into fields
* | parse-json content
  | project user_id, action, timestamp;

-- Parse and keep original JSON field
* | parse-json -keep raw_data;

-- Input: content: '{"user_id": 12345, "action": "login", "timestamp": "2025-10-16T10:30:00Z"}'
-- Output: user_id: '12345', action: 'login', timestamp: '2025-10-16T10:30:00Z'
```

--------------------------------

### Parse CSV Data

Source: https://context7.com/xingyunyang01/spl/llms.txt

Parse CSV-formatted text fields into separate columns. Supports custom delimiters and quote characters for flexible CSV parsing.

```SQL
-- Parse CSV line into named fields
* | parse-csv -delim=',' -quote='"' log_line as timestamp, level, module, message;

-- Parse with custom delimiter
* | parse-csv -delim='|' data as field1, field2, field3;

-- Input: log_line: '"2025-10-16 10:30:00","ERROR","Database","Connection failed"'
-- Output: timestamp: '2025-10-16 10:30:00', level: 'ERROR', module: 'Database', message: 'Connection failed'
```

--------------------------------

### Exclude Fields with 'project-away' in SPL

Source: https://github.com/xingyunyang01/spl/blob/main/字段操作指令.md

The 'project-away' command in SPL is used to remove fields that match a specified pattern. All other fields not matching the pattern are retained in the output. This command is useful for cleaning up data by excluding irrelevant fields.

```spl
| project-away -wildcard <field-pattern>, ...
```
Tokens:
Raw