### Install Rust and Set Environment

Source: https://docs.icegate.tech/llms-full.txt

Installs Rust using rustup and sets the environment variables. Verify the Rust version afterwards.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
rustc --version  # Should be >= 1.92.0
```

--------------------------------

### Install ANTLR JAR

Source: https://docs.icegate.tech/llms-full.txt

Installs the ANTLR Java Archive (JAR) file. This is a one-time setup step required for regenerating the LogQL parser.

```bash
make install

```

--------------------------------

### Complete Query Service Configuration Example

Source: https://docs.icegate.tech/llms-full.txt

This is a comprehensive example of a query.yaml configuration file. It demonstrates how to set up catalog backends, storage, engine parameters, and enable various observability features like Loki, Prometheus, Tempo, and Tracing.

```yaml
catalog:
  backend: !rest
    uri: http://nessie:19120/iceberg
  warehouse: s3://warehouse/
  properties:
    prefix: main
  cache:
    memory_size_mb: 1024
    disk_dir: /tmp/icegate/cache
    disk_size_mb: 4096

storage:
  backend: !s3
    bucket: warehouse
    region: us-east-1
    endpoint: http://minio:9000

engine:
  batch_size: 8192
  target_partitions: 4
  catalog_name: iceberg
  refresh_interval_secs: 15
  max_age_secs: 30
  wal_query_enabled: false
  wal_metadata_size_hint: 65536

queue:
  common:
    base_path: s3://queue/

loki:
  enabled: true
  host: 0.0.0.0
  port: 3100

prometheus:
  enabled: true
  host: 0.0.0.0
  port: 9090

tempo:
  enabled: true
  host: 0.0.0.0
  port: 3200

metrics:
  enabled: true
  host: 0.0.0.0
  port: 9091
  path: /metrics

tracing:
  enabled: true
  service_name: icegate-query
  otlp_endpoint: http://jaeger:4317
  sample_ratio: 1.0

```

--------------------------------

### Install IceGate from Local Charts

Source: https://docs.icegate.tech/llms-full.txt

Clone the IceGate repository and use this Helm command to install from local charts. This is useful for development or when direct registry access is not preferred.

```bash
git clone https://github.com/icegatetech/icegate.git
helm install icegate ./icegate/config/helm/icegate \
  --namespace icegate \
  --create-namespace \
  -f values.yaml
```

--------------------------------

### Start Development Environment

Source: https://docs.icegate.tech/llms-full.txt

Commands to launch the development environment using Skaffold or Docker Compose.

```bash
# Recommended: Skaffold with local Kubernetes
skaffold dev

# Alternative: Docker Compose with hot-reload
make dev
```

--------------------------------

### Install IceGate from OCI Registry

Source: https://docs.icegate.tech/llms-full.txt

Use this Helm command to install IceGate from the OCI registry. Ensure you have Helm 3 and Kubernetes 1.28+ installed.

```bash
helm install icegate oci://ghcr.io/icegatetech/charts/icegate \
  --version 0.1.0 \
  --namespace icegate \
  --create-namespace \
  -f values.yaml
```

--------------------------------

### Start IceGate Development Stack with Docker Compose

Source: https://docs.icegate.tech/llms-full.txt

Starts the core IceGate services with hot-reloading for development builds, or in release mode. Includes options for load generation, monitoring, and analytics.

```bash
# Core services with hot-reload (debug build)
make dev

# Core services in release mode
make run-core-release

# With load generator
make run-load-release

# With monitoring (Jaeger, Prometheus)
make run-monitoring-release

# With analytics (Trino SQL)
make run-analytics-release

# Stop all services
make down
```

--------------------------------

### Tracing Configuration Example

Source: https://docs.icegate.tech/llms-full.txt

Example configuration for exporting OpenTelemetry traces to Jaeger.

```yaml
tracing:
  enabled: true
  service_name: icegate-ingest
  otlp_endpoint: http://jaeger:4317
  sample_ratio: 0.1  # Sample 10% of traces in production
```

--------------------------------

### Development Environment Commands

Source: https://docs.icegate.tech/llms-full.txt

Commands for starting services and setting environment variables for local development.

```bash
# Start core services with hot-reload
make dev

# Start core services in release mode
make run-core-release

# Start with load generator
make run-load-release

# Start with monitoring (Jaeger, Prometheus, Grafana)
make run-analytics-release
```

```bash
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1
```

--------------------------------

### Install Skaffold on macOS and Linux

Source: https://docs.icegate.tech/llms-full.txt

Installs Skaffold using Homebrew on macOS or by downloading the binary on Linux.

```bash
# macOS
brew install skaffold

# Linux
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
chmod +x skaffold && sudo mv skaffold /usr/local/bin/
```

--------------------------------

### Development Environment Setup

Source: https://docs.icegate.tech/llms-full.txt

Instructions for setting up the local development environment using Docker Compose and environment variables.

```APIDOC
## Development Environment

For local development, use the provided Docker Compose configuration:

```bash
# Start core services with hot-reload
make dev

# Start core services in release mode
make run-core-release

# Start with load generator
make run-load-release

# Start with monitoring (Jaeger, Prometheus, Grafana)
make run-analytics-release
```

Environment variables for local development:

```bash
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1
```
```

--------------------------------

### Rust Documentation Example

Source: https://docs.icegate.tech/llms-full.txt

Example of documenting a public Rust function using doc comments. Includes arguments and return value descriptions.

```rust
/// Parses a LogQL query string into an AST.
///
/// # Arguments
///
/// * `query` - The LogQL query string
///
/// # Returns
///
/// The parsed LogQL expression or an error
pub fn parse(query: &str) -> Result<LogQLExpr> {
    // ...
}
```

--------------------------------

### Run IceGate with Skaffold

Source: https://docs.icegate.tech/llms-full.txt

Starts the IceGate development environment using Skaffold with different profiles for various configurations.

```bash
# Default profile (local k8s with MinIO + Nessie)
skaffold dev

# OrbStack profile
skaffold dev -p orbstack

# AWS Glue profile (pushes images to registry)
skaffold dev -p aws-glue

# External S3 profile
skaffold dev -p k3s-external-s3
```

--------------------------------

### Install Rust via rustup

Source: https://docs.icegate.tech/llms-full.txt

Installs Rust and Cargo using the official rustup script. This is a prerequisite for building the project from source.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

```

--------------------------------

### Run Core and Load Generator Services

Source: https://docs.icegate.tech/llms-full.txt

Starts the core IceGate services along with a load generator for testing purposes.

```bash
make run-load-release
```

--------------------------------

### Run Core, Analytics, and Monitoring Services

Source: https://docs.icegate.tech/llms-full.txt

Starts the core IceGate services, along with Trino for analytics and Jaeger, Prometheus, and Grafana for monitoring.

```bash
make run-analytics-release
```

--------------------------------

### Install OpenSSL on Ubuntu/Debian

Source: https://docs.icegate.tech/llms-full.txt

Installs OpenSSL development libraries and pkg-config on Ubuntu/Debian systems. This is necessary for resolving linking errors on these platforms.

```bash
apt install libssl-dev pkg-config

```

--------------------------------

### Verify IceGate Installation

Source: https://docs.icegate.tech/llms-full.txt

Check if pods are running in the 'icegate' namespace, port-forward to the query service, and test its readiness.

```bash
# Check pods are running
kubectl get pods -n icegate

# Port-forward to query service
kubectl port-forward -n icegate svc/icegate-query 3100:3100

# Test readiness
curl http://localhost:3100/ready
```

--------------------------------

### Install IceGate with Helm

Source: https://docs.icegate.tech/llms-full.txt

Commands to deploy IceGate using Helm charts, including options for custom values and storage configuration.

```bash
# Install from local charts
helm install icegate ./config/helm/icegate

# With custom values
helm install icegate ./config/helm/icegate \
  -f my-values.yaml \
  --set storage.bucket=my-warehouse
```

--------------------------------

### Verify Rust and Cargo Installation

Source: https://docs.icegate.tech/llms-full.txt

Checks if Rust and Cargo have been installed correctly by displaying their versions.

```bash
rustc --version
cargo --version

```

--------------------------------

### Instrument Go Services with OpenTelemetry

Source: https://docs.icegate.tech/llms-full.txt

Setup for the OpenTelemetry TracerProvider and OTLP gRPC exporter in Go.

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

exporter, _ := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("icegate-ingest:4317"),
    otlptracegrpc.WithInsecure(),
    otlptracegrpc.WithHeaders(map[string]string{
        "X-Scope-OrgID": "my-tenant",
    }),
)

res := resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceNameKey.String("api-gateway"),
)

tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(res),
)
otel.SetTracerProvider(tp)
```

--------------------------------

### Complete Ingest Service Configuration Example

Source: https://docs.icegate.tech/llms-full.txt

A comprehensive configuration for the Ingest service, including catalog, storage, queue, shift, and observability settings.

```yaml
catalog:
  backend: !rest
    uri: http://nessie:19120/iceberg
  warehouse: s3://warehouse/
  properties:
    prefix: main

storage:
  backend: !s3
    bucket: warehouse
    region: us-east-1
    endpoint: http://minio:9000

queue:
  common:
    base_path: s3://queue/
    channel_capacity: 1024
    max_row_group_size: 8192
  write:
    write_retries: 5
    compression: zstd
    records_per_flush_multiplier: 1
    max_bytes_per_flush: 67108864
    flush_interval_ms: 200

shift:
  read:
    max_record_batches_per_task: 1024
    max_input_bytes_per_task: 67108864
    plan_segment_read_parallelism: 8
    shift_segment_read_parallelism: 8
  write:
    row_group_size: 8192
    max_file_size_mb: 64
    table_cache_ttl_secs: 60
  jobsmanager:
    worker_count: 4
    poll_interval_ms: 1000
    iteration_interval_millisecs: 30000
    storage:
      endpoint: http://minio:9000
      bucket: jobs
      prefix: shifter
      region: us-east-1
      use_ssl: false
      job_state_codec: json
      request_timeout_secs: 5

otlp_http:
  enabled: true
  host: 0.0.0.0
  port: 4318

otlp_grpc:
  enabled: true
  host: 0.0.0.0
  port: 4317

metrics:
  enabled: true
  host: 0.0.0.0
  port: 9091
  path: /metrics

tracing:
  enabled: true
  service_name: icegate-ingest
  otlp_endpoint: http://jaeger:4317
  sample_ratio: 1.0
```

--------------------------------

### Explain Query Plan

Source: https://docs.icegate.tech/llms-full.txt

Use the Loki API to get an explanation of the query plan for a given LogQL query.

```bash
curl http://localhost:3100/loki/api/v1/explain \
  --data-urlencode 'query={service_name="api"}' \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Run Core Docker Compose Services

Source: https://docs.icegate.tech/llms-full.txt

Starts the core IceGate services including MinIO, Nessie, Ingest, Query, and Maintain using Docker Compose.

```bash
make run-core-release
```

--------------------------------

### Skaffold Development Command

Source: https://docs.icegate.tech/llms-full.txt

Run this command to start a development environment using Skaffold. Skaffold will manage the build and deployment process for local development.

```bash
skaffold dev
```

--------------------------------

### Basic Log Query with LogQL

Source: https://docs.icegate.tech/llms-full.txt

Query logs using LogQL against the Loki-compatible API on port 3100. This example retrieves logs for 'my-service' within the last hour.

```bash
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="my-service"}' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s 2>/dev/null || date -v-1H +%s) \
  --data-urlencode 'end='$(date +%s) \
  --data-urlencode 'limit=100' \
  -H "X-Scope-OrgID: demo"
```

--------------------------------

### GET /loki/api/v1/explain

Source: https://docs.icegate.tech/llms-full.txt

Retrieves the query execution plan for a given LogQL query.

```APIDOC
## GET /loki/api/v1/explain

### Description
Get query execution plan (IceGate extension).

### Method
GET

### Endpoint
/loki/api/v1/explain

### Parameters
#### Query Parameters
- **query** (string) - Required - LogQL query
```

--------------------------------

### Query Optimization and API Usage

Source: https://docs.icegate.tech/llms-full.txt

Examples for querying data, explaining execution plans, and limiting result sets using the Loki API.

```logql
# Good: timestamp range is specified via API parameters
{service_name="api"} |= "error"

# The Loki API start/end parameters drive partition pruning
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="api"}' \
  --data-urlencode 'start=1704067200' \
  --data-urlencode 'end=1704153600' \
  -H "X-Scope-OrgID: my-tenant"
```

```bash
curl -G http://localhost:3100/loki/api/v1/explain \
  --data-urlencode 'query={service_name="api"} |= "error"' \
  -H "X-Scope-OrgID: my-tenant"
```

```bash
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="api"}' \
  --data-urlencode 'limit=100' \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Install and Build IceGen for Load Testing

Source: https://docs.icegate.tech/llms-full.txt

Clone the IceGen repository and build the release binary. This tool is used for load testing IceGate ingestion with OpenTelemetry logs.

```bash
git clone https://github.com/icegatetech/icegen.git
cd icegen
cargo build --release
```

--------------------------------

### Configure Go OpenTelemetry SDK for Logs

Source: https://docs.icegate.tech/llms-full.txt

Initialize the Go OpenTelemetry SDK for log export using gRPC. This example shows how to create a new OTLP log exporter with the specified endpoint, insecure option, and tenant headers.

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"

exporter, _ := otlploggrpc.New(ctx,
    otlploggrpc.WithEndpoint("localhost:4317"),
    otlploggrpc.WithInsecure(),
    otlploggrpc.WithHeaders(map[string]string{
        "X-Scope-OrgID": "my-tenant",
    }),
)
```

--------------------------------

### IceGate Catalog Configuration

Source: https://docs.icegate.tech/llms-full.txt

Example YAML configuration for the IceGate catalog backend, specifying the Nessie REST URI and warehouse location.

```yaml
catalog:
  backend: !rest
    uri: http://nessie:19120/iceberg
  warehouse: s3://warehouse/
```

--------------------------------

### Ingest Logs for Multiple Teams

Source: https://docs.icegate.tech/llms-full.txt

Example of sending logs for different teams using distinct tenant identifiers.

```bash
# Platform team
curl -X POST http://localhost:4318/v1/logs \
  -H "X-Scope-OrgID: team-platform" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "gateway"}}]}, 
      "scopeLogs": [{"logRecords": [{"timeUnixNano": "1704067200000000000", "body": {"stringValue": "Request received"}, "severityText": "INFO", "severityNumber": 9}]}]
    }]
  }'

# Backend team
curl -X POST http://localhost:4318/v1/logs \
  -H "X-Scope-OrgID: team-backend" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "order-service"}}]}, 
      "scopeLogs": [{"logRecords": [{"timeUnixNano": "1704067200000000000", "body": {"stringValue": "Order created"}, "severityText": "INFO", "severityNumber": 9}]}]
    }]
  }'
```

--------------------------------

### Instrument Python Services with OpenTelemetry

Source: https://docs.icegate.tech/llms-full.txt

Setup for the OpenTelemetry TracerProvider and OTLP exporter in Python, including resource attributes and span creation.

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Configure tracing
resource = Resource.create({
    "service.name": "order-service",
    "service.version": "1.0.0",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="icegate-ingest:4317",
            headers={"X-Scope-OrgID": "my-tenant"},
            insecure=True,
        )
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("order-service")

# Create spans
with tracer.start_as_current_span("process-order") as span:
    span.set_attribute("order.id", "ORD-12345")
    span.set_attribute("http.method", "POST")
    span.set_attribute("http.status_code", 200)
    # ... business logic ...
```

--------------------------------

### Iceberg Table Partitioning Example

Source: https://docs.icegate.tech/llms-full.txt

Defines partitioning for Iceberg tables using identity partitioning on 'tenant_id', 'account_id', and 'day(timestamp)'. This pattern is common for multi-tenancy.

```sql
partitioning = ARRAY["tenant_id", "account_id", "day(timestamp)"]
```

--------------------------------

### Get Query Execution Plan (Loki API Explain)

Source: https://docs.icegate.tech/llms-full.txt

Provides the query execution plan for a given LogQL query. This is an IceGate extension. Requires the X-Scope-OrgID header.

```bash
curl -G http://localhost:3100/loki/api/v1/explain \
  --data-urlencode 'query=count_over_time({service_name="api-service"}[5m])' \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Short-Term Operational Retention Strategy (7 Days)

Source: https://docs.icegate.tech/llms-full.txt

Example SQL commands for daily execution to delete logs and spans older than 7 days, and to expire snapshots and remove orphan files.

```sql
-- Run daily via cron or scheduled job
DELETE FROM icegate.logs WHERE timestamp < NOW() - INTERVAL '7' DAY;
DELETE FROM icegate.spans WHERE start_timestamp < NOW() - INTERVAL '7' DAY;
```

```sql
ALTER TABLE icegate.logs EXECUTE expire_snapshots(retention_threshold => '1d');
ALTER TABLE icegate.spans EXECUTE expire_snapshots(retention_threshold => '1d');
```

```sql
ALTER TABLE icegate.logs EXECUTE remove_orphan_files(retention_threshold => '1d');
ALTER TABLE icegate.spans EXECUTE remove_orphan_files(retention_threshold => '1d');
```

--------------------------------

### Loki API Instant Query Example

Source: https://docs.icegate.tech/llms-full.txt

This curl command demonstrates how to perform an instant query against the Loki API to count log entries for a specific service within a 5-minute window. Ensure the X-Scope-OrgID header is set for authentication.

```bash
curl -G http://localhost:3100/loki/api/v1/query \
  --data-urlencode 'query=count_over_time({service_name="api-service"}[5m])' \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Complete IceGate Data Source Provisioning

Source: https://docs.icegate.tech/llms-full.txt

Deploy all three IceGate data sources (Loki, Tempo, Prometheus) in a single Grafana provisioning file. This example sets Loki as the default data source and configures trace-to-log linking.

```yaml
apiVersion: 1
datasources:
  - name: IceGate Logs
    type: loki
    access: proxy
    url: http://icegate-query:3100
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: default
    isDefault: true
    uid: icegate-loki

  - name: IceGate Traces
    type: tempo
    access: proxy
    url: http://icegate-query:3200
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      tracesToLogs:
        datasourceUid: icegate-loki
        tags: ['service.name']
        mappedTags: [{ key: 'service.name', value: 'service_name' }]
        mapTagNamesEnabled: true
        filterByTraceID: true
    secureJsonData:
      httpHeaderValue1: default
    uid: icegate-tempo

  - name: IceGate Metrics
    type: prometheus
    access: proxy
    url: http://icegate-query:9090
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: default
    uid: icegate-prometheus
```

--------------------------------

### Build and Run Project

Source: https://docs.icegate.tech/llms-full.txt

Standard commands for cloning, building, and testing the repository.

```bash
# Clone the repository
git clone https://github.com/icegatetech/icegate.git
cd icegate

# Build the project
cargo build

# Run tests
cargo test
```

--------------------------------

### Find Matching Series with curl

Source: https://docs.icegate.tech/llms-full.txt

Search for log series that match a given label selector. This example finds series where 'service_name' starts with 'my-'. Requires the 'demo' X-Scope-OrgID.

```bash
curl -G http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service_name=~"my-.*"}' \
  -H "X-Scope-OrgID: demo"
```

--------------------------------

### LogQL: Select Logs using Label Regex Matching

Source: https://docs.icegate.tech/llms-full.txt

Use regex matching with LogQL to select log streams where the service name follows a pattern. This example matches any service name starting with 'api-'.

```logql
{service_name=~"api-.*"}
```

--------------------------------

### Serve User Documentation Locally

Source: https://docs.icegate.tech/llms-full.txt

Serve the user documentation locally for previewing changes. Navigate to the docs directory and run the serve script.

```bash
cd docs && npm run serve
```

--------------------------------

### LogQL: Select Logs by Multiple Labels

Source: https://docs.icegate.tech/llms-full.txt

Combine multiple label selectors in LogQL to narrow down log streams. This example selects logs from 'api-service' with a severity of 'ERROR'.

```logql
{service_name="api-service", severity_text="ERROR"}
```

--------------------------------

### LogQL: Identify Top Services by Log Volume

Source: https://docs.icegate.tech/llms-full.txt

Determine the services generating the most log data by summing byte rates grouped by service name in LogQL. This example uses a 5-minute interval for logs from the 'app' job.

```logql
sum by (service_name) (bytes_rate({job="app"}[5m]))
```

--------------------------------

### Search Traces using curl

Source: https://docs.icegate.tech/llms-full.txt

Example using curl to search for traces with specific tags, minimum duration, and a limit. Uses URL encoding for parameters and requires the `X-Scope-OrgID` header.

```bash
curl -G http://localhost:3200/api/search \
  --data-urlencode 'tags=service.name=api-service' \
  --data-urlencode 'minDuration=100ms' \
  --data-urlencode 'limit=10' \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Send logs via gRPC with multiple tenants and concurrency

Source: https://docs.icegate.tech/llms-full.txt

Use this command to send logs using gRPC transport with specified tenant count and concurrency. Ensure the otel-log-generator is installed and configured.

```bash
otel-log-generator otel \
  --endpoint http://localhost:4317 \
  --transport grpc \
  --tenant-count 8 \
  --count 1000 \
  --concurrency 20
```

--------------------------------

### Aggregate Logs into Metrics with LogQL (Count)

Source: https://docs.icegate.tech/llms-full.txt

Aggregate logs into metrics using LogQL's `count_over_time` function. This example counts logs for 'my-service' in 5-minute intervals over the last hour.

```bash
# Count logs per 5-minute window
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query=count_over_time({service_name="my-service"}[5m])' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s 2>/dev/null || date -v-1H +%s) \
  --data-urlencode 'end='$(date +%s) \
  --data-urlencode 'step=300' \
  -H "X-Scope-OrgID: demo"
```

--------------------------------

### LogQL: Filter Logs by Bytes Label Comparison

Source: https://docs.icegate.tech/llms-full.txt

Filter log streams by byte size labels using LogQL. This example selects logs where the 'bytes' label is greater than 1KB.

```logql
{service_name="api-service"} | bytes > 1KB
```

--------------------------------

### Send Logs via OTLP gRPC (Python SDK)

Source: https://docs.icegate.tech/llms-full.txt

Configure the OpenTelemetry Python SDK to send logs via OTLP gRPC. This example sets up a LoggerProvider with a BatchLogRecordProcessor and an OTLPLogExporter pointing to localhost:4317.

```python
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

provider = LoggerProvider()
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint="localhost:4317",
            headers={"X-Scope-OrgID": "demo"},
            insecure=True,
        )
    )
)
```

--------------------------------

### Install OpenSSL on macOS

Source: https://docs.icegate.tech/llms-full.txt

Installs OpenSSL using Homebrew on macOS. This is a prerequisite for resolving linking errors on macOS that depend on OpenSSL libraries.

```bash
brew install openssl

```

--------------------------------

### Create and Apply Custom Overlay

Source: https://docs.icegate.tech/llms-full.txt

Copy an existing overlay and modify its configuration files for a new environment. Apply the custom overlay using kubectl.

```bash
cp -r config/kustomize/overlays/orbstack config/kustomize/overlays/my-env
vi config/kustomize/overlays/my-env/values-icegate.yaml
vi config/kustomize/overlays/my-env/secret-aws.yaml
kubectl apply -k config/kustomize/overlays/my-env
```

--------------------------------

### Run Query Service

Source: https://docs.icegate.tech/llms-full.txt

Executes the query service binary. Requires a configuration file specified with the `-c` flag.

```bash
cargo run --bin query -- run -c config/docker/query.yaml

```

--------------------------------

### Run Query Service with Configuration

Source: https://docs.icegate.tech/llms-full.txt

Execute the query service using a specified YAML configuration file.

```bash
query run -c /etc/icegate/query.yaml
```

--------------------------------

### LogQL: Sum Log Counts by Service

Source: https://docs.icegate.tech/llms-full.txt

Aggregate log counts across different services using LogQL's `sum by` clause. This example sums the counts over 5-minute intervals for logs associated with the 'app' job.

```logql
sum by (service_name) (count_over_time({job="app"}[5m]))
```

--------------------------------

### GET /ready

Source: https://docs.icegate.tech/llms-full.txt

Performs a health check on the service.

```APIDOC
## GET /ready

### Description
Health check endpoint.

### Method
GET

### Endpoint
/ready

### Response
#### Success Response (200)
- **status** (string) - Status of the service

#### Response Example
{
  "status": "ready"
}
```

--------------------------------

### Run Maintain Service (Create Migration)

Source: https://docs.icegate.tech/llms-full.txt

Executes the maintain service binary to create a new database migration. Requires a configuration file specified with the `-c` flag.

```bash
cargo run --bin maintain -- migrate create -c config/docker/maintain.yaml

```

--------------------------------

### GET /loki/api/v1/query_range

Source: https://docs.icegate.tech/llms-full.txt

Query logs or metrics over a time range.

```APIDOC
## GET /loki/api/v1/query_range

### Description
Query logs or metrics over a time range.

### Method
GET

### Endpoint
/loki/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - LogQL query
- **start** (int) - Required - Start timestamp (Unix seconds or nanoseconds)
- **end** (int) - Required - End timestamp (Unix seconds or nanoseconds)
- **limit** (int) - Optional - Maximum number of entries (default: 100)
- **step** (duration) - Optional - Query resolution step (e.g., "5m")
- **direction** (string) - Optional - forward or backward (default: backward)

### Request Example
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="api-service"}' \
  --data-urlencode 'start=1704067200' \
  --data-urlencode 'end=1704153600' \
  --data-urlencode 'limit=1000' \
  -H "X-Scope-OrgID: my-tenant"

### Response
#### Success Response (200)
- **status** (string) - Status of the request
- **data** (object) - Result data containing resultType and result

#### Response Example
{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [
      {
        "stream": {
          "service_name": "api-service",
          "severity_text": "INFO"
        },
        "values": [
          ["1704067200000000000", "Request processed successfully"]
        ]
      }
    ]
  }
}
```

--------------------------------

### GET /api/v1/query_range

Source: https://docs.icegate.tech/llms-full.txt

Queries metrics over a specified time range using PromQL.

```APIDOC
## GET /api/v1/query_range

### Description
Query metrics over a time range.

### Method
GET

### Endpoint
/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - PromQL query
- **start** (float) - Required - Start timestamp (Unix seconds)
- **end** (float) - Required - End timestamp (Unix seconds)
- **step** (duration) - Required - Query resolution step
```

--------------------------------

### GET /health - Health Check

Source: https://docs.icegate.tech/llms-full.txt

Check the health status of the IceGate service.

```APIDOC
## GET /health

### Description
Check the health status of the IceGate service.

### Method
GET

### Endpoint
/health

### Response
#### Success Response (200)
- **status** (string) - The health status of the service (e.g., "healthy").

#### Response Example
```json
{
  "status": "healthy"
}
```
```

--------------------------------

### GET /api/traces/{trace_id}

Source: https://docs.icegate.tech/llms-full.txt

Retrieves all spans associated with a specific trace ID.

```APIDOC
## GET /api/traces/{trace_id}

### Description
Retrieves all spans for a given trace ID.

### Method
GET

### Endpoint
http://localhost:3200/api/traces/{trace_id}

### Parameters
#### Path Parameters
- **trace_id** (string) - Required - The unique identifier for the trace.

#### Request Headers
- **X-Scope-OrgID** (string) - Required - Tenant identifier.
```

--------------------------------

### GET /loki/api/v1/series

Source: https://docs.icegate.tech/llms-full.txt

Retrieves label sets matching specific log stream selectors.

```APIDOC
## GET /loki/api/v1/series

### Description
Get label sets matching selectors.

### Method
GET

### Endpoint
/loki/api/v1/series

### Parameters
#### Query Parameters
- **match[]** (string) - Required - Log stream selector(s)
- **start** (int) - Optional - Start timestamp
- **end** (int) - Optional - End timestamp

### Response
#### Success Response (200)
- **status** (string) - Status of the request
- **data** (array) - List of matching label sets

#### Response Example
{
  "status": "success",
  "data": [
    {"service_name": "api-service", "severity_text": "INFO"},
    {"service_name": "api-gateway", "severity_text": "ERROR"}
  ]
}
```

--------------------------------

### GET /loki/api/v1/label/{name}/values

Source: https://docs.icegate.tech/llms-full.txt

Retrieves the values associated with a specific label in Loki.

```APIDOC
## GET /loki/api/v1/label/{name}/values

### Description
Get values for a specific label.

### Method
GET

### Endpoint
/loki/api/v1/label/{name}/values

### Parameters
#### Path Parameters
- **name** (string) - Required - The label name

#### Query Parameters
- **start** (int) - Optional - Start timestamp
- **end** (int) - Optional - End timestamp

### Response
#### Success Response (200)
- **status** (string) - Status of the request
- **data** (array) - List of label values

#### Response Example
{
  "status": "success",
  "data": [
    "api-service",
    "worker-service",
    "gateway"
  ]
}
```

--------------------------------

### Apply Kustomize Overlays

Source: https://docs.icegate.tech/llms-full.txt

Command to apply specific Kustomize configurations for different deployment environments.

```bash
# Apply with kustomize
kubectl apply -k config/kustomize/overlays/aws-glue
```

--------------------------------

### GET /loki/api/v1/labels

Source: https://docs.icegate.tech/llms-full.txt

Retrieves a list of all available label names within the Loki system.

```APIDOC
## GET /loki/api/v1/labels

### Description
Get all label names.

### Method
GET

### Endpoint
/loki/api/v1/labels

### Parameters
#### Query Parameters
- **start** (int) - Optional - Start timestamp
- **end** (int) - Optional - End timestamp

### Response
#### Success Response (200)
- **status** (string) - Status of the request
- **data** (array) - List of label names

#### Response Example
{
  "status": "success",
  "data": [
    "service_name",
    "severity_text",
    "trace_id"
  ]
}
```

--------------------------------

### Maintain Service Configuration

Source: https://docs.icegate.tech/llms-full.txt

Configuration for the Maintain service, requiring only catalog and storage setup.

```APIDOC
## Maintain Service Configuration

The Maintain service only requires catalog and storage configuration:

```yaml
catalog:
  backend: !rest
    uri: http://nessie:19120/iceberg
  warehouse: s3://warehouse/
  properties:
    prefix: main

storage:
  backend: !s3
    bucket: warehouse
    region: us-east-1
    endpoint: http://minio:9000
```
```

--------------------------------

### GET /loki/api/v1/query_range

Source: https://docs.icegate.tech/llms-full.txt

Queries logs within a specific time range using Loki API.

```APIDOC
## GET /loki/api/v1/query_range

### Description
Queries logs based on a query string and time range.

### Method
GET

### Endpoint
http://localhost:3100/loki/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - The log query expression.
- **start** (integer) - Required - Start timestamp (Unix epoch).
- **end** (integer) - Required - End timestamp (Unix epoch).
- **step** (integer) - Optional - Query step size.

### Request Headers
- **X-Scope-OrgID** (string) - Required - Tenant identifier.
```

--------------------------------

### Ingest with Account-Level Partitioning

Source: https://docs.icegate.tech/llms-full.txt

Include both tenant and account headers to support granular data partitioning.

```bash
curl -X POST http://localhost:4318/v1/logs \
  -H "X-Scope-OrgID: tenant-123" \
  -H "X-Account-ID: account-456" \
  -H "Content-Type: application/json" \
  -d '...'
```

--------------------------------

### Get Label Values with curl

Source: https://docs.icegate.tech/llms-full.txt

Retrieve all values for a specific label, such as 'service_name'. Requires the 'demo' X-Scope-OrgID.

```bash
curl http://localhost:3100/loki/api/v1/label/service_name/values \
  -H "X-Scope-OrgID: demo"
```

--------------------------------

### Trace Response Structure

Source: https://docs.icegate.tech/llms-full.txt

Example JSON response structure for a trace query, showing batches, resource information, and spans.

```json
{
  "batches": [
    {
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "api-service"}}
        ]
      },
      "scopeSpans": [
        {
          "spans": [
            {
              "traceId": "5B8EFFF798038103D269B633813FC60C",
              "spanId": "EEE19B7EC3C1B174",
              "name": "GET /api/users",
              "kind": 2,
              "startTimeUnixNano": "1704067200000000000",
              "endTimeUnixNano": "1704067200100000000",
              "status": {"code": 1}
            }
          ]
        }
      ]
    }
  ]
}
```

--------------------------------

### Build IceGate (Release)

Source: https://docs.icegate.tech/llms-full.txt

Compiles the project in release mode, optimized for production. Build artifacts will be located in the `target/release/` directory.

```bash
cargo build --release

```

--------------------------------

### Ingest Data for Tenant 'team-a' with curl

Source: https://docs.icegate.tech/llms-full.txt

Example of ingesting logs for a specific tenant ('team-a') using the 'X-Scope-OrgID' header.

```bash
# Ingest for tenant "team-a"
curl -X POST http://localhost:4318/v1/logs \
  -H "X-Scope-OrgID: team-a" \
  -H "Content-Type: application/json" \
  -d '...'
```

--------------------------------

### Configure MinIO WAL Lifecycle Rule

Source: https://docs.icegate.tech/llms-full.txt

Set a 1-day Time-To-Live (TTL) on the queue bucket for WAL segments in MinIO.

```bash
# Set 1-day TTL on queue bucket
mc ilm rule add --expire-days 1 myminio/queue
```

--------------------------------

### Run All Tests

Source: https://docs.icegate.tech/llms-full.txt

Execute all unit and integration tests within the project to verify functionality.

```bash
cargo test
```

--------------------------------

### Configure Server and Routes

Source: https://docs.icegate.tech/llms-full.txt

Manages shared state, graceful shutdown via CancellationToken, and router configuration.

```rust
#[derive(Clone)]
pub struct ModuleState {
    pub resource: Arc<SharedResource>,
}

pub async fn run(
    resource: Arc<SharedResource>,
    config: ModuleConfig,
    cancel_token: CancellationToken,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let addr: SocketAddr = format!("{}:{}", config.host, config.port).parse()?;
    let state = ModuleState { resource };
    let app = super::routes::routes(state);

    let listener = tokio::net::TcpListener::bind(addr).await?;
    axum::serve(listener, app)
        .with_graceful_shutdown(async move { cancel_token.cancelled().await })
        .await?;
    Ok(())
}
```

```rust
pub fn routes(state: ModuleState) -> Router {
    Router::new()
        .route("/v1/endpoint", post(handlers::endpoint_handler))
        .route("/health", get(handlers::health))
        .with_state(state)
}
```

--------------------------------

### Get Trace by ID

Source: https://docs.icegate.tech/llms-full.txt

Retrieves a complete trace using its unique trace ID. Requires the X-Scope-OrgID header for tenant identification.

```APIDOC
## GET /api/traces/{traceID}

### Description
Retrieve a complete trace by its trace ID.

### Method
GET

### Endpoint
/api/traces/{traceID}

### Parameters
#### Path Parameters
- **traceID** (string) - Required - 32-character hex trace ID

#### Request Headers
- **X-Scope-OrgID** (string) - Required - Tenant identifier (e.g., `my-tenant`)

### Response
#### Success Response (200)
- **batches** (array) - Contains resource and scope spans for the trace.

#### Response Example
```json
{
  "batches": [
    {
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "api-service"}}
        ]
      },
      "scopeSpans": [
        {
          "spans": [
            {
              "traceId": "5B8EFFF798038103D269B633813FC60C",
              "spanId": "EEE19B7EC3C1B174",
              "name": "GET /api/users",
              "kind": 2,
              "startTimeUnixNano": "1704067200000000000",
              "endTimeUnixNano": "1704067200100000000",
              "status": {"code": 1}
            }
          ]
        }
      ]
    }
  ]
}
```
```

--------------------------------

### Get Label Values (Prometheus API)

Source: https://docs.icegate.tech/llms-full.txt

Retrieves values for a specific label name from the Prometheus-compatible API. Requires the X-Scope-OrgID header.

```bash
curl http://localhost:9090/api/v1/label/service_name/values \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Run Maintain Service with Configuration

Source: https://docs.icegate.tech/llms-full.txt

Execute the maintain service, including schema migration commands, using a specified YAML configuration file.

```bash
# Schema migration create
maintain migrate create -c /etc/icegate/maintain.yaml
# Schema migration upgrade
maintain migrate upgrade -c /etc/icegate/maintain.yaml
```

--------------------------------

### Configure In-Memory Storage (Testing)

Source: https://docs.icegate.tech/llms-full.txt

Use the in-memory storage backend, suitable for testing purposes.

```yaml
storage:
  backend: !memory
```

--------------------------------

### Build Specific IceGate Binaries

Source: https://docs.icegate.tech/llms-full.txt

Compiles only a specific binary (e.g., query, ingest, maintain) instead of the entire project. This can speed up build times when working on a particular service.

```bash
# Query service only
cargo build --bin query

# Ingest service only
cargo build --bin ingest

# Maintain service only
cargo build --bin maintain

```

--------------------------------

### Limit Time Range for Query Performance

Source: https://docs.icegate.tech/llms-full.txt

Specify 'start' and 'end' timestamps in query parameters to limit the time range and improve performance.

```bash
--data-urlencode 'start=1704067200'
--data-urlencode 'end=1704153600'
```

--------------------------------

### Get All Label Names (Prometheus API)

Source: https://docs.icegate.tech/llms-full.txt

Retrieves all available label names from the Prometheus-compatible API. Requires the X-Scope-OrgID header for tenant identification.

```bash
curl http://localhost:9090/api/v1/labels \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Run Ingest Service with Configuration

Source: https://docs.icegate.tech/llms-full.txt

Execute the ingest service using a specified YAML configuration file.

```bash
ingest run -c /etc/icegate/ingest.yaml
```

--------------------------------

### Horizontal Scaling for Ingest Service

Source: https://docs.icegate.tech/llms-full.txt

Example Helm values for horizontally scaling the Ingest service by increasing replica count and adjusting resource requests/limits.

```yaml
# Helm values.yaml
ingest:
  replicaCount: 3
  resources:
    requests:
      cpu: "2"
      memory: 4Gi
    limits:
      cpu: "4"
      memory: 8Gi
```

--------------------------------

### Build User Documentation

Source: https://docs.icegate.tech/llms-full.txt

Build the user documentation using Diplodoc (YFM Markdown) by navigating to the docs directory and running the build script.

```bash
cd docs && npm run build
```

--------------------------------

### Query Data for Tenant 'team-a' with curl

Source: https://docs.icegate.tech/llms-full.txt

Example of querying logs for a specific tenant ('team-a'), demonstrating data isolation via the 'X-Scope-OrgID' header.

```bash
# Query only sees team-a's data
curl -G http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="api"}' \
  -H "X-Scope-OrgID: team-a"
```

--------------------------------

### Build Query Service Docker Image

Source: https://docs.icegate.tech/llms-full.txt

Builds the Docker image for the query service in release mode, specifying the binary and profile.

```bash
docker build -t icegate/query:latest \
  --build-arg BINARY=query \
  --build-arg PROFILE=release \
  -f config/docker/Dockerfile .
```

--------------------------------

### Build Docker Image for Dev

Source: https://docs.icegate.tech/llms-full.txt

Builds a Docker image for the query service in development mode with debug profiling.

```bash
docker build -t icegate/query:dev \
  --build-arg BINARY=query \
  --build-arg PROFILE=debug \
  -f config/docker/Dockerfile .
```

--------------------------------

### Perform Health Check via HTTP

Source: https://docs.icegate.tech/llms-full.txt

This cURL command checks the health status of the IceGate service by sending a GET request to the /health endpoint.

```bash
curl http://localhost:4318/health
```

--------------------------------

### Get Tag Values

Source: https://docs.icegate.tech/llms-full.txt

Fetch all unique values for a specific tag, such as 'service.name'. This is useful for exploring the different services or attributes present in your trace data.

```bash
curl http://localhost:3200/api/search/tag/service.name/values \
  -H "X-Scope-OrgID: my-tenant"
```

--------------------------------

### Iceberg Table Timestamp Precision Example

Source: https://docs.icegate.tech/llms-full.txt

Specifies microsecond precision for all timestamps, including timezone information. This ensures high-resolution time tracking in observability data.

```sql
TIMESTAMP(6) WITH TIME ZONE
```