### curl GET request to list rules Source: https://observatorium.io/docs/design/rules-api.md Example curl command to retrieve the raw rules for a tenant. ```bash curl http:///api/metrics/v1//api/v1/rules/raw ``` -------------------------------- ### Clone repository and enter local example directory Source: https://observatorium.io/docs/usage/getting-started.md Clone the Observatorium repository and change into the local example directory. This is the first step before deploying the stack locally. ```bash git clone https://github.com/observatorium/observatorium.git cd configuration/examples/local ``` -------------------------------- ### Authorizer configuration example (YAML) Source: https://observatorium.io/docs/proposals/drafts/20211106-plugable-authorizers.md Example configuration for an authorizer using the OPA type, including query, paths to policy files, and rate limits. ```yaml - name: test-oidc id: 1610b0c3-c509-4592-a256-a1871353dbfa authorizer: type: opa config: query: data.observatorium.allow paths: - ./test/config/observatorium.rego - ./test/config/rbac.yaml rateLimits: - endpoint: "/api/metrics/v1/.+/api/v1/receive" limit: 100 window: 1s - endpoint: "/api/logs/v1/.*" limit: 100 window: 1s ``` -------------------------------- ### Run quickstart script to deploy Observatorium Source: https://observatorium.io/docs/usage/getting-started.md Run the quickstart script to deploy Observatorium using Kubernetes manifests generated from Jsonnet. The script sets up everything automatically. ```bash ./quickstart.sh ``` -------------------------------- ### Example response for create rules Source: https://observatorium.io/docs/design/rules-api.md Response body returned on successful creation of rules. ```text successfully updated rules file ``` -------------------------------- ### GenerateWithMimic Configuration Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md Example usage of the generator to create Observatorium API Kubernetes configuration with various options. ```go // Observatorium API with no sidecar and a serviceMonitor. generator.GenerateWithMimic( g, api.NewObservatoriumAPI( api.WithLogLevel("debug"), api.WithMetrics( "http://observatorium-xyz-thanos-query-frontend.observatorium.svc.cluster.local:9090", "http://observatorium-xyz-thanos-receive.observatorium.svc.cluster.local:19291", "http://observatorium-xyz-rules-objstore.observatorium.svc.cluster.local:8080", ), api.WithLogs( "http://observatorium-xyz-loki-query-frontend-http.observatorium.svc.cluster.local:3100", "http://observatorium-xyz-loki-distributor-http.observatorium.svc.cluster.local:3100", "http://observatorium-xyz-loki-ruler-http.observatorium.svc.cluster.local:3100", "http://observatorium-xyz-loki-querier-http.observatorium.svc.cluster.local:3100", ), api.WithTraces( "http://observatorium-xyz-jaeger-query.observatorium.svc.cluster.local:16686/", "observatorium-xyz-otel-collector:4317", ), api.WithRateLimiter("observatorium-xyz-gubernator.observatorium.svc.cluster.local:8081"), api.WithRBACYAML(string(rbacData)), api.WithTenantsSecret(map[string]string{"tenants.yaml": string(tenantsData)}), api.WithImage("quay.io/observatorium/api", "main-2023-01-24-v0.1.2-318-g5f4fdf4"), api.WithName("observatorium-xyz"), api.WithNamespace("observatorium"), api.WithReplicas(3), api.WithAPIResources(apiResources), api.WithServiceMonitor(), ).Manifests(), "config", ) ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/labels Source: https://observatorium.io/docs/api.html Retrieves all labels for a specified tenant. The tenant is provided as a path parameter, and optional start and end timestamps can be supplied as query parameters. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/labels ### Description Get all labels for a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/labels ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Query Parameters - **start** (string) - Optional - Start timestamp (RFC3339 or Unix timestamp) - **end** (string) - Optional - End timestamp (RFC3339 or Unix timestamp) ### Response #### Success Response (2XX) Labels for a tenant. #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": ["string"] } #### Error Responses - **4XX**: Bad request - **5XX**: Server side error while getting labels ``` -------------------------------- ### GET /api/v1/rules/raw Source: https://observatorium.io/docs/design/rules-api.md Lists the rules for a tenant. The response format is in application/yaml. ```APIDOC ## GET /api/v1/rules/raw ### Description Lists the rules for a tenant. The response format is in application/yaml. ### Method GET ### Endpoint /api/v1/rules/raw ### Parameters #### Query Parameters - **tenant** (string) - Required - The tenant name. ### Request Example ``` curl http:///api/metrics/v1//api/v1/rules/raw ``` ### Response #### Success Response (200) - **body** (application/yaml) - Successfully listed rules. #### Response Example ``` groups: - interval: 30s name: test-alerting-rule rules: - alert: HighRequestLatency annotations: summary: High request latency expr: job:request_latency_seconds:mean5m{job="myjob",tenant_id="1234"} > 0.5 for: 10m labels: severity: page tenant_id: 1234 ``` ### Error Handling - **401** - Error finding tenant/tenant ID. - **500** - A server side error happened while trying to list rules. ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/label/{name}/values Source: https://observatorium.io/docs/api.html Retrieves the values for a specific label of a tenant. The tenant and label name are provided as path parameters, and optional start and end timestamps can be supplied as query parameters. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/label/{name}/values ### Description Get label values for a label of a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/label/{name}/values ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant - **name** (string) - Required - Label name to get values for #### Query Parameters - **start** (string) - Optional - Start timestamp (RFC3339 or Unix timestamp) - **end** (string) - Optional - End timestamp (RFC3339 or Unix timestamp) ### Response #### Success Response (2XX) Label values for a tenant. #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "string", "data": ["node"] } #### Error Responses - **4XX**: Bad request - **5XX**: Server side error while getting label values ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/query Source: https://observatorium.io/docs/api.html Evaluates an instant PromQL query for a tenant. Returns the query result for the specified tenant. ```APIDOC ## [GET] /api/metrics/v1/{tenant}/api/v1/query ### Description Evaluate instant queries for a tenant. You can evaluate instant queries for a tenant. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/query ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **query** (string) - Optional - PromQL query to fetch result for metrics - **timeout** (string) - Optional - Evaluation timeout - **dedup** (boolean) - Optional - Query deduplication (Thanos) - **partial_response** (boolean) - Optional - Query partial response (Thanos) - **time** (string) - Optional - Evaluation timestamp ### Response #### Success Response (2XX) instance query result for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "resultType": "matrix", "result": [ { "metric": { "property1": "string", "property2": "string" }, "values": [["string"]] } ] } } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/query Source: https://observatorium.io/docs/api.html Evaluates an instant LogQL query for a specified tenant. Returns the query result in JSON format. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/query ### Description Evaluates an instant LogQL query for a specified tenant. Returns the query result in JSON format. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/query ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Query Parameters - **query** (string) - Required - LogQL query to fetch result for logs - **limit** (number) - Optional - Max number of entries - **time** (string) - Optional - Evaluation timestamp (rfc3339 or unix_timestamp) - **direction** (string) - Optional - Determines the sort order of logs. Supported values are forward or backward. Defaults to backward. ### Response #### Success Response (2XX) Instance query result for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "resultType": "matrix", "result": [ { "metric": { "property1": "string", "property2": "string" }, "values": [["string"]] } ] } } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error while getting queries ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/series Source: https://observatorium.io/docs/api.html Get series for a tenant. Returns a JSON object with a data array of label sets. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/series ### Description Get series for a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/series ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Query Parameters - **match[]** (Array of strings) - Required - Repeated series selector argument - **start** (string ) - Optional - Start timestamp - **end** (string ) - Optional - End timestamp ### Response #### Success Response (2XX) Series for a tenant #### Response Example ```json { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": [ { "property1": "string", "property2": "string" } ] } ``` #### Error Responses - **4XX** - Bad request - **5XX** - Server side error while getting series ``` -------------------------------- ### GET /api/v1/rules/raw - List rules endpoint Source: https://observatorium.io/docs/design/rules-api.md HTTP method and path for listing rules for a tenant. Requires tenant-scoped authentication token. ```http GET /api/v1/rules/raw ``` -------------------------------- ### Example response for list rules Source: https://observatorium.io/docs/design/rules-api.md YAML response listing the tenant's rules. The tenant_id label is enforced in the read path, appearing in labels and in the expression. ```yaml groups: - interval: 30s name: test-alerting-rule rules: - alert: HighRequestLatency annotations: summary: High request latency expr: job:request_latency_seconds:mean5m{job="myjob",tenant_id="1234"} > 0.5 for: 10m labels: severity: page tenant_id: 1234 ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/tail Source: https://observatorium.io/docs/api.html Lists log messages based on query. You can get log messages based on query. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/tail ### Description Lists log messages based on query. You can get log messages based on query. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/tail ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **start** (string) - Optional - Start timestamp (rfc3339 or unix_timestamp) - **query** (string) - Optional - LogQL query to fetch result for logs - **delay** (integer) - Optional - delay retrieving logs ### Responses - **2XX** - logs for a tenant - **4XX** - Bad request - **5XX** - Server side error while tailing logs ### Response Example (2XX) ```json { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "streams": [ { "stream": { "property1": "string", "property2": "string" }, "values": [["string"]] } ], "dropped_entries": [ { "labels": { "property1": "string", "property2": "string" }, "timestamp": "string" } ] } ``` ``` -------------------------------- ### Example rule.yaml file with alerting rule Source: https://observatorium.io/docs/design/rules-api.md YAML rule file following the Observatorium OpenAPI specification, based on Prometheus rules syntax. Contains an alerting rule for high request latency. ```yaml groups: - name: test-alerting-rule interval: 30s rules: - alert: HighRequestLatency expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/rules/raw Source: https://observatorium.io/docs/api.html Lists all configured rules for a tenant in YAML form. This endpoint retrieves the raw rules configuration for a specified tenant. ```APIDOC ## GET /api/metrics/v1/{tenant}/api/v1/rules/raw ### Description Lists all configured rules for a tenant in YAML form. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/rules/raw ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant ### Responses - **200** - all rules for a tenant - **401** - error finding tenant ID - **403** - unauthorized - **500** - server side error while listing rules ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/series Source: https://observatorium.io/docs/api.html Retrieves series data for a specified tenant. This endpoint accepts match[] query parameters to filter series and optional start and end timestamps. ```APIDOC ## GET /api/metrics/v1/{tenant}/api/v1/series ### Description Get series for a tenant. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/series ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Query Parameters - **match[]** (Array of strings) - Required - Repeated series selector argument - **start** (string) - Optional - Start timestamp (RFC3339 or Unix timestamp) - **end** (string) - Optional - End timestamp (RFC3339 or Unix timestamp) ### Response #### Success Response (2XX) Series for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": [ { "property1": "string", "property2": "string" } ] } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### Tenant configuration with OIDC authenticator Source: https://observatorium.io/docs/proposals/drafts/20211007-pluggable-authenticators.md Example tenant configuration for the side-effect import approach, defining an OIDC authenticator with client credentials, issuer settings, and rate limits. ```yaml - name: test-oidc id: 1610b0c3-c509-4592-a256-a1871353dbfa authenticator: type: oidc config: clientID: test clientSecret: xyz issuerCAPath: ./tmp/certs/ca.pem issuerURL: http://127.0.0.1:5556/dex redirectURL: https://localhost:8443/oidc/test-oidc/callback usernameClaim: email rateLimits: - endpoint: "/api/metrics/v1/.+/api/v1/receive" limit: 100 window: 1s - endpoint: "/api/logs/v1/.*" limit: 100 window: 1s ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/rules Source: https://observatorium.io/docs/api.html Retrieves all logs rules for a tenant in YAML form for all known namespaces and groups. Requires the tenant name as a path parameter. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/rules ### Description Get all logs rules for a tenant in YAML form for all known namespaces and groups. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/rules ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant ### Response #### Success Response (2XX) All logs rules for a tenant #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} Source: https://observatorium.io/docs/api.html Retrieves all logs rules in a specific namespace for a tenant in YAML form. Requires tenant and namespace path parameters. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} ### Description Get all logs rules in a namespace for a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant - **namespace** (string) - Required - namespace of the log rule group ### Response #### Success Response (2XX) All rule groups in a namespace for a tenant #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/rules Source: https://observatorium.io/docs/api.html Gets rules for a tenant in JSON form with health data. This endpoint retrieves the rules configuration along with health information. ```APIDOC ## GET /api/metrics/v1/{tenant}/api/v1/rules ### Description Get Rules for a tenant in JSON form. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/rules ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **type** (string) - Optional - Enum: "alert" "record" - type of Rules - **match[]** (Array of strings) - Optional - Repeated label selector argument ### Responses - **2XX** - rule for a tenant - **4XX** - Bad request - **5XX** - Server side error ### Response Example ```json { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "groups": [ { "interval": 0, "name": "telemeter.rules", "file": "rules.yaml", "limit": 0, "evaluationTime": 0.002900887, "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "partialResponseStrategy": "WARN", "rules": [ { "query": "count by (name) (cluster{condition=\"halted\"} == 1)", "name": "count:cluster_halted", "health": "ok", "evaluationTime": 0.002900887, "lastError": "string", "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "type": "recording", "labels": { "property1": "string", "property2": "string" } } ] } ] } } ``` ``` -------------------------------- ### GET /api/logs/v1/{tenant}/prometheus/api/v1/rules Source: https://observatorium.io/docs/api.html Retrieves all alerting and recording rules currently loaded for a tenant in JSON format. The tenant is specified as a path parameter. ```APIDOC ## GET /api/logs/v1/{tenant}/prometheus/api/v1/rules ### Description Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded for a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/prometheus/api/v1/rules ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant ### Response #### Success Response (2XX) All rules for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": [ "string" ], "status": "success", "data": { "groups": [ { "interval": 0, "name": "telemeter.rules", "file": "rules.yaml", "limit": 0, "evaluationTime": 0.002900887, "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "partialResponseStrategy": "WARN", "rules": [ { "query": "count by (name) (cluster{condition=\"halted\"} == 1)", "name": "count:cluster_halted", "health": "ok", "evaluationTime": 0.002900887, "lastError": "string", "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "type": "recording", "labels": { "property1": "string", "property2": "string" } } ] } ] } } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### Get series for a tenant endpoint Source: https://observatorium.io/docs/api.html Endpoint path for retrieving series for a specific tenant, used to handle large numbers of stream selectors. ```text post/api/logs/v1/{tenant}/loki/api/v1/series ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/query_range Source: https://observatorium.io/docs/api.html Evaluates a range PromQL query for a tenant. Returns the range query result for the specified tenant. ```APIDOC ## [GET] /api/metrics/v1/{tenant}/api/v1/query_range ### Description Evaluate range queries for a tenant. You can evaluate range queries for a tenant. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/query_range ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **query** (string) - Optional - PromQL query to fetch result for metrics - **start** (string) - Optional - Start timestamp - **end** (string) - Optional - End timestamp - **timeout** (string) - Optional - Evaluation timeout - **dedup** (boolean) - Optional - Query deduplication (Thanos) - **partial_response** (boolean) - Optional - Query partial response (Thanos) - **step** (string) - Optional - Query resolution step width ### Response #### Success Response (2XX) range query result for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "resultType": "matrix", "result": [ { "metric": { "property1": "string", "property2": "string" }, "values": [["string"]] } ] } } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### JSON response sample for tenant rules Source: https://observatorium.io/docs/api.html Example JSON response structure for the rules retrieval endpoint, showing rule groups, health status, and evaluation metrics. ```json { "error": "string", "errorType": "string", "warnings": [ "string" ], "status": "success", "data": { "groups": [ { "interval": 0, "name": "telemeter.rules", "file": "rules.yaml", "limit": 0, "evaluationTime": 0.002900887, "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "partialResponseStrategy": "WARN", "rules": [ { "query": "count by (name) (cluster{condition=\"halted\"} == 1)", "name": "count:cluster_halted", "health": "ok", "evaluationTime": 0.002900887, "lastError": "string", "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "type": "recording", "labels": { "property1": "string", "property2": "string" } } ] } ] } } ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/query_range Source: https://observatorium.io/docs/api.html Evaluates a range LogQL query for a specified tenant. Returns the range query result in JSON format. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/query_range ### Description Evaluates a range LogQL query for a specified tenant. Returns the range query result in JSON format. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/query_range ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Query Parameters - **query** (string) - Required - LogQL query to fetch result for logs - **start** (string) - Optional - Start timestamp (rfc3339 or unix_timestamp) - **end** (string) - Optional - End timestamp (rfc3339 or unix_timestamp) - **limit** (number) - Optional - Max number of entries - **step** (string) - Optional - Query resolution step width - **interval** (string) - Optional - Only return entries at (or >) the specified interval. Only applies to queries which produce a stream response. - **direction** (string) - Optional - Determines the sort order of logs. Supported values are forward or backward. Defaults to backward. ### Response #### Success Response (2XX) Range query result for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "resultType": "matrix", "result": [ { "metric": { "property1": "string", "property2": "string" }, "values": [["string"]] } ] } } #### Error Responses - **4XX** - Bad request - **5XX** - Server side error while getting range queries ``` -------------------------------- ### Get rules for a tenant in JSON Source: https://observatorium.io/docs/api.html Endpoint to retrieve rules for a tenant in JSON format, including health data and optional query parameters for type and label selectors. ```text get/api/metrics/v1/{tenant}/api/v1/rules ``` -------------------------------- ### curl PUT request to create rules Source: https://observatorium.io/docs/design/rules-api.md Example curl command to upload a YAML rule file to the raw rules endpoint. The request overwrites all existing rules for the tenant. ```bash curl -X PUT --data-binary @alerting-rule.yaml --header "Content-Type: application/yaml" http:///api/metrics/v1//api/v1/rules/raw ``` -------------------------------- ### GET /api/logs/v1/{tenant}/prometheus/api/v1/alerts Source: https://observatorium.io/docs/api.html Prometheus-compatible rules endpoint to list all active alerts for a tenant. Returns a JSON object with alert groups and rules. ```APIDOC ## GET /api/logs/v1/{tenant}/prometheus/api/v1/alerts ### Description Prometheus-compatible rules endpoint to list all active alerts for a tenant. ### Method GET ### Endpoint /api/logs/v1/{tenant}/prometheus/api/v1/alerts ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant ### Response #### Success Response (2XX) All active alerts for a tenant #### Response Example ```json { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": { "groups": [ { "interval": 0, "name": "telemeter.rules", "file": "rules.yaml", "limit": 0, "evaluationTime": 0.002900887, "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "partialResponseStrategy": "WARN", "rules": [ { "query": "count by (name) (cluster{condition=\"halted\"} == 1)", "name": "count:cluster_halted", "health": "ok", "evaluationTime": 0.002900887, "lastError": "string", "lastEvaluation": "2022-02-20T15:32:12.759781322Z", "type": "recording", "labels": { "property1": "string", "property2": "string" } } ] } ] } } ``` #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### GET /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace}/{group} Source: https://observatorium.io/docs/api.html Retrieves a specific log rule group in a namespace for a tenant in YAML form. Returns the rule group on success. ```APIDOC ## GET /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace}/{group} ### Description Get the logs Rule group in a namespace for a tenant in YAML form. ### Method GET ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace}/{group} ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant - **namespace** (string) - Required - Namespace of the log rule group - **group** (string) - Required - Group of the log rule ### Response #### Success Response (2XX) Rule group for a tenant #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### POST /api/logs/v1/{tenant}/loki/api/v1/series Source: https://observatorium.io/docs/api.html Gets series for a tenant. Used when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits. ```APIDOC ## POST /api/logs/v1/{tenant}/loki/api/v1/series ### Description Gets series for a tenant. Used when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits. ### Method POST ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/series ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **match[]** (Array of strings) - Required - Repeated series selector argument - **start** (string) - Optional - Start timestamp (rfc3339 or unix_timestamp) - **end** (string) - Optional - End timestamp (rfc3339 or unix_timestamp) #### Request Body (application/x-www-form-urlencoded) - **match[]** (Array of strings) - Optional - Repeated series selector argument - **start** (string) - Optional - Start timestamp (rfc3339 or unix_timestamp) - **end** (string) - Optional - End timestamp (rfc3339 or unix_timestamp) ### Responses - **201** - successfully writes series to tenant - **401** - error finding tenant ID - **403** - unauthorized - **500** - server side error while writing series ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/label/{label_name}/values Source: https://observatorium.io/docs/api.html Retrieves the values for a specific label of a tenant. This endpoint requires the tenant and label name as path parameters and supports optional query parameters for filtering. ```APIDOC ## GET /api/metrics/v1/{tenant}/api/v1/label/{label_name}/values ### Description Get label values for a tenant. You can get label values for a label of a tenant. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/label/{label_name}/values ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant - **label_name** (string) - Required - label name to get values for #### Query Parameters - **match[]** (Array of strings) - Optional - Repeated series selector argument - **start** (string ) - Optional - Start timestamp - **end** (string ) - Optional - End timestamp ### Response #### Success Response (2XX) label values for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "string", "data": ["node"] } #### Error Responses - **4XX**: Bad request - **5XX**: Server side error ``` -------------------------------- ### GET /api/metrics/v1/{tenant}/api/v1/labels Source: https://observatorium.io/docs/api.html Retrieves all label names for a specified tenant. This endpoint accepts optional series selector and time range query parameters to filter the labels. ```APIDOC ## GET /api/metrics/v1/{tenant}/api/v1/labels ### Description Get all labels for a tenant. You can get labels for a tenant. ### Method GET ### Endpoint /api/metrics/v1/{tenant}/api/v1/labels ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant #### Query Parameters - **match[]** (Array of strings) - Optional - Repeated series selector argument - **start** (string ) - Optional - Start timestamp - **end** (string ) - Optional - End timestamp ### Response #### Success Response (2XX) labels for a tenant #### Response Example { "error": "string", "errorType": "string", "warnings": ["string"], "status": "success", "data": ["string"] } #### Error Responses - **4XX**: Bad request - **5XX**: Server side error ``` -------------------------------- ### obsctl Deployment Workflow Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md A proposed three-step deployment process using obsctl to generate boilerplate and apply manifests. ```bash obsctl gen observatorium-default go main.go generate --output=manifests kubect apply -f manifests/ ``` -------------------------------- ### List rules for a tenant in YAML Source: https://observatorium.io/docs/api.html Endpoint to list all configured rules for a specific tenant in YAML format. ```text get/api/metrics/v1/{tenant}/api/v1/rules/raw ``` -------------------------------- ### NewObservatoriumAPI Constructor Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md Initializes a new instance of the Observatorium API with customizable options, returning essential Kubernetes objects. ```go // NewObservatoriumAPI returns a new instance of Observatorium API, customized with options. // Also includes options for adding sidecars. // Returns the following K8s Objects: // Deployment // Service // ServiceAccount // ConfigMap // Secret func NewObservatoriumAPI(opts ...ObservatoriumAPIOption) *observatoriumAPI { ``` -------------------------------- ### POST /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} Source: https://observatorium.io/docs/api.html Creates and updates rule groups in a namespace for a tenant. Requires tenant and namespace path parameters and a YAML request body. ```APIDOC ## POST /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} ### Description Set logs rule groups in a namespace for a tenant. You can create and update rule groups in a namespace for a tenant. ### Method POST ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/rules/{namespace} ### Parameters #### Path Parameters - **tenant** (string) - Required - name of the tenant - **namespace** (string) - Required - namespace of the log rule group #### Request Body Schema: application/yaml - **name** (string) - Description not provided - **interval** (string) - Description not provided - **rules[]** (Array of RecordingRule (object) or AlertingRule (object)) - Description not provided ### Response #### Success Response (2XX) Successfully created and/or updated rule groups in a namespace for a tenant #### Error Responses - **4XX** - Bad request - **5XX** - Server side error ``` -------------------------------- ### PUT /api/v1/rules/raw - Create rules endpoint Source: https://observatorium.io/docs/design/rules-api.md HTTP method and path for setting rules for a tenant. Requires tenant-scoped authentication token in the Authorization header. ```http PUT /api/v1/rules/raw ``` -------------------------------- ### Manifests Method Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md Generates Kubernetes YAML manifests as a map of names to runtime objects, allowing for manual overrides if necessary. ```go // Manifests creates resultant K8s YAML manifests for customized Observatorium API deployment // as a map, containing manifest name and runtime.Object. // This provides the ability to override options of the returned manifests // in a specific manner (in case the exported ObservatoriumAPIOption functions do not suffice). func (c *observatoriumAPI) Manifests(opts ...ObservatoriumAPIOption) k8sutil.ObjectMap { ``` -------------------------------- ### WithSidecars Option Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md Allows overriding default Kubernetes deployment settings and adding additional sidecar containers. ```go // WithSidecars allows overriding default K8s Deployment and adding additional containers, // alongside Observatorium API. // Use SidecarConfig struct to attach additonal volumes, service ports and service monitor // scrape endpoints for these sidecars. func WithSidecars(s k8sutil.SidecarConfig) ObservatoriumAPIOption { return func(a *observatoriumAPI) { a.sidecars = s } } ``` -------------------------------- ### Instant Query Response Sample Source: https://observatorium.io/docs/api.html The JSON structure returned by the /api/logs/v1/{tenant}/loki/api/v1/query endpoint upon a successful 2XX response. ```json { * "error": "string", * "errorType": "string", * "warnings": [ * "string" ], * "status": "success", * "data": { * "resultType": "matrix", * "result": [ * { * "metric": { * "property1": "string", * "property2": "string" }, * "values": [ * [ * "string" ] ] } ] } } ``` -------------------------------- ### WithMetrics Option Source: https://observatorium.io/docs/proposals/20230130-go-configuration.md Enables metrics signals for the Observatorium API by providing read, write, and rules URLs as flags. ```go // WithMetrics enables metrics signal for Observatorium API and includes passed in URL args as flags. // If any of the URLs are empty, the flags are not included. func WithMetrics(metricsRead, metricsWrite, metricsRules string) ObservatoriumAPIOption { return func(a *observatoriumAPI) { a.metricsRead = metricsRead a.metricsWrite = metricsWrite a.metricsRules = metricsRules } } ``` -------------------------------- ### Architecture diagram: Jaeger per-tenant deployment with OpenTelemetry collector Source: https://observatorium.io/docs/proposals/approved/20201125-jaeger-traces-store.md ASCII architecture diagram showing data flow from Observatorium API to OpenTelemetry collector, then to per-tenant Jaeger collectors, and to Elasticsearch. The Observatorium API also routes directly to per-tenant Jaeger Query components. Illustrates the hard multitenancy approach where each tenant uses dedicated collector and query components. ```text +--------------------+ | | +-------------------->| Jaeger collector | | | tenant1 | +-----------------------+ +-------------+-------------+ +----------+---------+ +-----------------+ | | | | | | | | Observatorium API +-------->| OpenTelemetry collector | +------------------> Elasticsearch | | | | | | | | +-----------+-----------+ +-------------+-------------+ +----------+---------+ +--------^--------+ | | | | | | +-------------------->| Jaeger collector | | | | tenant2 | | | +--------------------+ | | | | +----------------+ | | | | | +---------------------------------------------------------->| Jaeger Query +-------------------+ | | tenant1 | | | +----------------+ | | | | +----------------+ | | | | | +---------------------------------------------------------->| Jaeger Query +-------------------+ | tenant2 | +----------------+ ``` -------------------------------- ### POST /api/logs/v1/{tenant}/loki/api/v1/push Source: https://observatorium.io/docs/api.html Sends log entries to Loki for a specified tenant. The request body must contain a stream object and an array of values representing the log entries. ```APIDOC ## POST /api/logs/v1/{tenant}/loki/api/v1/push ### Description Sends log entries to Loki. ### Method POST ### Endpoint /api/logs/v1/{tenant}/loki/api/v1/push ### Parameters #### Path Parameters - **tenant** (string) - Required - Name of the tenant #### Request Body - **stream** (object) - Required - Stream object containing key-value pairs - **values** (Array of strings or strings) - Required - Array of log entry values ### Request Example { "stream": { "property1": "string", "property2": "string" }, "values": [ ["string"] ] } ### Response #### Success Response (204) Writes all log entries of corresponding tenant to Loki #### Error Responses - **401** - Error finding tenant ID - **403** - Unauthorized - **500** - Server side error while pushing logs ``` -------------------------------- ### Set or overwrite rules for a tenant Source: https://observatorium.io/docs/api.html Endpoint to set or overwrite rules for a specific tenant using an application/yaml request body. ```text put/api/metrics/v1/{tenant}/api/v1/rules/raw ``` -------------------------------- ### Proposed OPA Authorizer Payload Source: https://observatorium.io/docs/proposals/drafts/20210709-opa-authorizer-labels-observatorium-api.md The proposed extension adds an optional 'matchers' field containing a list of label matchers to refine read access. ```json { "result": true|false, "matchers": [ { "name": "label key", "type": "MatchEqual|MatchNotEqual|MatchRegex|MatchNotReqex", "value": "label value" } ] } ```