### Download setup script and example config Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/aws?q= Downloads the setup script and example configuration file for the multi-account AWS MCP setup. Run these commands before proceeding to the configuration step. ```bash curl -O https://raw.githubusercontent.com/robusta-dev/holmes-mcp-integrations/master/servers/aws/setup-multi-account-iam.sh chmod +x setup-multi-account-iam.sh # Download example configuration file curl -O https://raw.githubusercontent.com/robusta-dev/holmes-mcp-integrations/master/servers/aws/multi-cluster-config-example.yaml ``` -------------------------------- ### Download multi-account setup script and example config Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/aws Downloads the multi-account setup script and an example configuration file from the Holmes MCP integrations repository. Make the script executable before running it. ```bash # Download the setup script curl -O https://raw.githubusercontent.com/robusta-dev/holmes-mcp-integrations/master/servers/aws/setup-multi-account-iam.sh chmod +x setup-multi-account-iam.sh # Download example configuration file curl -O https://raw.githubusercontent.com/robusta-dev/holmes-mcp-integrations/master/servers/aws/multi-cluster-config-example.yaml ``` -------------------------------- ### Quick start with AWS Bedrock Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Quick start with AWS Bedrock. Set up AWS credentials, install boto3, create a test pod, and ask your first question using a Bedrock model. See AWS Bedrock Configuration for more details. ```bash export AWS_ACCESS_KEY_ID="your-access-key" export AWS_SECRET_ACCESS_KEY="your-secret-key" export AWS_DEFAULT_REGION="your-region" ``` ```bash pip install "boto3>=1.28.57" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0" # Or use another model holmes ask "what is wrong with the user-profile-import pod?" --model="bedrock/" ``` -------------------------------- ### Install and start HolmesGPT HTTP server with Docker Compose Source: https://holmesgpt.dev/latest/installation/docker-compose-installation Steps to install and start the HolmesGPT HTTP server with Docker Compose: clone the repository, set the OpenAI API key, start the server, and verify it is running. ```bash git clone https://github.com/HolmesGPT/holmesgpt.git cd holmesgpt ``` ```bash export OPENAI_API_KEY="your-api-key" ``` ```bash docker compose up ``` ```bash curl http://localhost:5050/healthz ``` -------------------------------- ### Example usage: list tables and sizes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-postgresql Prompt-style example asking for a list of all tables and their sizes. ```text "List all tables and their sizes" ``` -------------------------------- ### Example usage: get table schema Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-postgresql Prompt-style example asking HolmesGPT for the schema of the users table. ```text "Show me the schema for the users table" ``` -------------------------------- ### List tables prompt Source: https://holmesgpt.dev/latest/data-sources/database-connectors?q= Example prompt asking Holmes to list all tables and their sizes in the prod-postgres toolset. Use this to get an overview of the database. ```text "List all tables and their sizes in prod-postgres" ``` -------------------------------- ### Run multi-account setup script Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/aws?q= Commands to run the multi-account setup script. Supports basic setup, custom config, custom permissions file, verification, and teardown. The teardown removes all created resources. ```bash # Basic usage (uses default config: multi-cluster-config.yaml) ./setup-multi-account-iam.sh setup # With custom config file ./setup-multi-account-iam.sh setup my-config.yaml # With custom permissions file ./setup-multi-account-iam.sh setup my-config.yaml ./aws-mcp-iam-policy.json # Verify the setup ./setup-multi-account-iam.sh verify my-config.yaml # Teardown (removes all created resources) ./setup-multi-account-iam.sh teardown my-config.yaml ``` -------------------------------- ### Install Azure API MCP server with go install Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp?q= Install the Azure API MCP server using Go 1.24+ and rename the installed binary to azure-api-mcp for clarity. The binary is placed in $GOPATH/bin. ```bash go install github.com/Azure/azure-api-mcp/cmd/server@latest ``` ```bash mv "$(go env GOPATH)/bin/server" "$(go env GOPATH)/bin/azure-api-mcp" ``` -------------------------------- ### Quick start with Google Gemini Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Quick start with Google Gemini. Set up the API key and model environment variables, then create a test pod to investigate. The TOOL_SCHEMA_NO_PARAM_OBJECT_IF_NO_PARAMS environment variable is required for Gemini. ```bash export GEMINI_API_KEY="your-gemini-api-key" export TOOL_SCHEMA_NO_PARAM_OBJECT_IF_NO_PARAMS=true export MODEL="gemini/" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` -------------------------------- ### Apply and verify Azure MCP instance Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp?q= Applies the manifest and verifies the deployment with kubectl get pods and logs. Uses the example name prod and namespace monitoring. ```bash kubectl apply -f azure-mcp-additional-instance.yaml # verify it comes up (use the name/namespace you chose in Step 2) kubectl get pods -n monitoring -l app=azure-prod kubectl logs -n monitoring -l app=azure-prod ``` -------------------------------- ### Quick start with OpenAI Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Quick start with OpenAI. Set the API key, create a test pod, and ask your first question. The default model is gpt-4.1; you can specify a different model with --model. ```bash export OPENAI_API_KEY="your-api-key" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash # Uses gpt-4.1 by default (fairly fast, decent results) holmes ask "what is wrong with the user-profile-import pod?" # Or specify a different model holmes ask "what is wrong with the user-profile-import pod?" --model="gpt-5" ``` -------------------------------- ### Install Dependencies Source: https://holmesgpt.dev/latest/operator/development Commands to install Python dependencies, activate the virtual environment, and install pre-commit hooks using Poetry. ```bash # Install Python dependencies with Poetry poetry install # Activate virtual environment poetry shell # Install pre-commit hooks poetry run pre-commit install ``` -------------------------------- ### Install HolmesGPT from Source Source: https://holmesgpt.dev/latest/installation/cli-installation Clone the repository and install dependencies using Poetry for development builds. ```bash git clone https://github.com/HolmesGPT/holmesgpt.git cd holmesgpt poetry install --no-root ``` ```bash poetry run holmes ask --help ``` -------------------------------- ### Example response for /api/model Source: https://holmesgpt.dev/latest/reference/http-api?q= Example response for the /api/model GET request, showing a list of model names. ```json { "model_name": ["anthropic/claude-sonnet-4-5-20250929", "anthropic/claude-opus-4-5-20251101", "robusta"] } ``` -------------------------------- ### Ollama quick start: create test pod, ask HolmesGPT with ollama_chat Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Create a test pod and ask HolmesGPT to investigate it using the ollama_chat model. No API key is required for a local Ollama installation. Ollama can be tricky to configure; trying a hosted model first is recommended. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="ollama_chat/" ``` -------------------------------- ### Vertex AI quick start: set credentials, create test pod, ask HolmesGPT Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Set up Google Vertex AI credentials, create a test pod, and ask HolmesGPT to investigate it using the vertex_ai model. The commands are part of the Vertex AI quick start workflow. ```bash export VERTEXAI_PROJECT="your-project-id" export VERTEXAI_LOCATION="us-central1" export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account-key.json" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="vertex_ai/" ``` -------------------------------- ### Run setup script for GCP service account Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/gcp?q= Recommended setup script that grants ~50 read-only roles and skips key generation. Provides audit logs, networking, security metadata, and monitoring access with read-only boundaries. ```bash git clone https://github.com/robusta-dev/holmes-mcp-integrations.git cd holmes-mcp-integrations/servers/gcp ./setup-gcp-service-account.sh --project ${PROJECT_ID} --skip-key-generation ``` -------------------------------- ### Multi-account configuration YAML Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/aws?q= Example configuration file for multi-account setup. Defines clusters, Kubernetes namespace/service account, IAM role/policy names, and target accounts with profiles. Edit this file with your cluster details and account IDs. ```yaml clusters: - name: prod-cluster region: us-east-1 account_id: "111111111111" oidc_issuer_id: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA oidc_issuer_url: https://oidc.eks.us-east-1.amazonaws.com/id/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA - name: staging-cluster region: us-west-2 account_id: "111111111111" oidc_issuer_id: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB oidc_issuer_url: https://oidc.eks.us-west-2.amazonaws.com/id/BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB kubernetes: namespace: YOUR_NAMESPACE # Must match the namespace where Holmes is deployed service_account: multi-account-mcp-sa iam: role_name: EKSMultiAccountMCPRole policy_name: MCPReadOnlyPolicy session_duration: 3600 target_accounts: - profile: dev account_id: "111111111111" description: "Development account" - profile: prod account_id: "222222222222" description: "Production account" ``` -------------------------------- ### Holmes OSS CLI: Configure skill_repos in config.yaml and run holmes ask Source: https://holmesgpt.dev/latest/reference/skills?q= For Holmes OSS CLI: add the skill repo to ~/.holmes/config.yaml with GitHub App credentials, then export the private key and run the holmes CLI. Holmes syncs the repo at the start of each run, minting an installation token from the private key. ```yaml skill_repos: - url: https://github.com//.git branch: main # optional; defaults to the repo's default branch sub_path: skills # optional; subdirectory inside the repo where SKILL.md dirs live github_app_id: "" github_app_installation_id: "" github_app_private_key_env: GITHUB_APP_PRIVATE_KEY ``` ```bash export GITHUB_APP_PRIVATE_KEY="$(cat /path/to/private-key.pem)" holmes ask "why is my pod crashing?" ``` -------------------------------- ### Install HolmesGPT via Homebrew Source: https://holmesgpt.dev/latest/installation/cli-installation Steps to add the tap, install the package, upgrade, and verify the installation. ```bash brew tap robusta-dev/homebrew-holmesgpt ``` ```bash brew install holmesgpt ``` ```bash brew upgrade holmesgpt ``` ```bash holmes ask --help ``` -------------------------------- ### Quick start with Azure AI Foundry Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Quick start with Azure AI Foundry. Set the API key and endpoint environment variables, create a test pod, and ask your first question using the Azure model. See Azure AI Foundry Configuration for more details. ```bash export AZURE_API_VERSION="2024-02-15-preview" export AZURE_API_BASE="https://your-resource.openai.azure.com" export AZURE_API_KEY="your-azure-api-key" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="azure/" ``` -------------------------------- ### Inline toolsets configuration example Source: https://holmesgpt.dev/latest/development/evaluations/adding-evals Inline toolsets configuration example showing how to enable kubernetes/core and disable aws/cloudwatch. This is the first method described under Advanced Features > Toolsets Configuration. ```yaml toolsets: kubernetes/core: enabled: true aws/cloudwatch: enabled: false ``` -------------------------------- ### Example usage: check missing indexes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-postgresql Prompt-style example asking for missing indexes on the orders table. ```text "Check for missing indexes on the orders table" ``` -------------------------------- ### Example prompt: show table schema Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-postgresql?q= Example user prompt asking for the schema of a specific table. Useful for understanding table structure. ```text "Show me the schema for the users table" ``` -------------------------------- ### Install and verify vela CLI Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubevela Install the KubeVela CLI using the official script and verify the installation with `vela version`. ```bash # Install vela CLI curl -fsSl https://kubevela.io/script/install.sh | bash # Verify installation vela version ``` -------------------------------- ### Example Usage for Datadog Traces Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/datadog Example commands to query Datadog traces using the Holmes CLI. ```bash # Find slow requests holmes ask "find traces where the checkout service took longer than 5 seconds" # Analyze specific trace holmes ask "analyze trace ID abc123 for performance issues" # Service dependencies holmes ask "show me traces involving both payment and inventory services" ``` -------------------------------- ### Enable API server HTTPS (HOLMES_SSL_*) Source: https://holmesgpt.dev/latest/reference/environment-variables?q= Enable HTTPS for the HolmesGPT API. The CLI example sets the HOLMES_SSL_* environment variables; the Helm example enables TLS with a Kubernetes secret. Setting both HOLMES_SSL_CERTFILE and HOLMES_SSL_KEYFILE enables HTTPS; setting only one, or setting HOLMES_SSL_CA_CERTS or HOLMES_SSL_KEYFILE_PASSWORD without both server-side files, causes the server to fail to start. The certificate is read once at startup and does not hot-reload. ```bash export HOLMES_SSL_CERTFILE=/path/to/tls.crt export HOLMES_SSL_KEYFILE=/path/to/tls.key # optional: export HOLMES_SSL_KEYFILE_PASSWORD=changeit export HOLMES_SSL_CA_CERTS=/path/to/ca.crt # enables mTLS ``` ```yaml tls: enabled: true secretName: holmes-tls # secret with tls.crt and tls.key # caCertsSecretKey: ca.crt # optional: key in the secret, enables mTLS ``` -------------------------------- ### Context Window Override Example Source: https://holmesgpt.dev/latest/ai-providers/other Example of using custom_args.max_context_size to manually set the context window to 1M tokens. ```yaml modelList: my-model: model: provider/model-name api_key: "{{ env.API_KEY }}" custom_args: max_context_size: 1000000 # Override to 1M tokens ``` -------------------------------- ### Example Usage for Datadog Metrics Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/datadog Example commands to query Datadog metrics using the Holmes CLI. ```bash # List available metrics holmes ask "what metrics are available for my application?" # Query CPU usage holmes ask "show me CPU usage for the payment service over the last 6 hours" # Custom application metrics holmes ask "analyze the payment_processing_time metric for anomalies" ``` -------------------------------- ### Example Datadog Traces Queries Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/datadog?q= Example usage of the Datadog traces toolset. Shows how to query for slow requests, analyze a specific trace, and find service dependencies using the holmes ask command. ```bash # Find slow requests holmes ask "find traces where the checkout service took longer than 5 seconds" # Analyze specific trace holmes ask "analyze trace ID abc123 for performance issues" # Service dependencies holmes ask "show me traces involving both payment and inventory services" ``` -------------------------------- ### Install HolmesGPT via pipx Source: https://holmesgpt.dev/latest/installation/cli-installation Install HolmesGPT using pipx and verify the installation. ```bash pipx install holmesgpt ``` ```bash holmes ask --help ``` -------------------------------- ### Command approval prompt example Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/bash Example of the interactive prompt displayed when a command is not in the allow list. The user can choose to run once, approve the prefix, or provide feedback. ```text Bash command kubectl scale deployment nginx --replicas=3 Scale nginx deployment to 3 replicas Do you want to proceed? 1. Yes 2. Yes, and don't ask again for `kubectl scale deployment nginx` commands 3. Type here to tell Holmes what to do differently ``` -------------------------------- ### Example MySQL usage: analyze slow query Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-mysql Literal prompt example for the toolset: analyze a slow query on orders. ```text "Analyze slow query: SELECT * FROM orders WHERE created_at > '2024-01-01'" ``` -------------------------------- ### Install boto3 with minimum version Source: https://holmesgpt.dev/latest/ai-providers/aws-bedrock?q= Install boto3 version 1.28.57 or higher, required for AWS Bedrock integration. ```bash pip install "boto3>=1.28.57" ``` -------------------------------- ### List collections and schemas Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/mongodb Example prompt for listing all collections in the orders database and displaying their schemas. ```text "List all collections in the orders database and show their schemas" ``` -------------------------------- ### Ask Questions Without Kubernetes Source: https://holmesgpt.dev/latest/installation/cli-installation Examples of querying HolmesGPT about infrastructure when not using a Kubernetes demo pod. ```bash holmes ask "what Prometheus alerts are currently firing and why?" holmes ask "what is the health of my Elasticsearch cluster?" holmes ask "are there any issues with my production databases?" ``` -------------------------------- ### Common Use Case: Suggest Indexes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlite?q= Example query to suggest indexes for a table. ```sql "Suggest indexes for the orders table" ``` -------------------------------- ### Example response for /api/admin/reload/toolsets Source: https://holmesgpt.dev/latest/reference/http-api?q= Example response for the /api/admin/reload/toolsets POST request, showing status and counts of loaded toolsets and skills. ```json { "status": "ok", "component": "toolsets", "detail": "50 toolsets loaded, 15 enabled, 12 skills", "counts": { "toolsets_total": 50, "toolsets_enabled": 15, "skills": 12 } } ``` -------------------------------- ### Example response for /api/admin/reload Source: https://holmesgpt.dev/latest/reference/http-api?q= Example response for the /api/admin/reload POST request, showing status and counts of loaded components. ```json { "status": "ok", "component": "all", "detail": "50 toolsets (15 enabled), 12 skills, 4 models", "counts": { "toolsets_total": 50, "toolsets_enabled": 15, "skills": 12, "models_loaded": 4 } } ``` -------------------------------- ### Example NRQL Query for Application Logs Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/newrelic An example of an automatically constructed NRQL query used by Holmes when investigating application logs. ```sql SELECT message, timestamp FROM Log WHERE pod_name = 'your-app' SINCE 1 hour ago ``` -------------------------------- ### Start interactive shell with /shell Source: https://holmesgpt.dev/latest/reference/slash-commands Starts an interactive shell session. When you exit, you can share the entire session with the AI and add a comment or question. ```text > /shell $ kubectl logs failing-pod-xyz $ kubectl describe pod failing-pod-xyz $ exit Share shell session with AI? (Y/n): y Add a comment or question (optional): The pod keeps failing with OOM errors ``` -------------------------------- ### Start the Docker Compose server Source: https://holmesgpt.dev/latest/installation/docker-compose-installation?q= Starts the HolmesGPT HTTP server using Docker Compose. The server will be available at http://localhost:5050. ```bash docker compose up ``` -------------------------------- ### Example HolmesGPT database prompts Source: https://holmesgpt.dev/latest/data-sources/database-connectors Example natural-language prompts for HolmesGPT that use the database tools. Each prompt references a configured database instance (prod-postgres or analytics-clickhouse) and asks for a query, schema, table listing, or index check. ```text "Why is this query slow on prod-postgres: SELECT * FROM orders WHERE status = 'pending'" ``` ```text "Show me the schema for the users table in analytics-clickhouse" ``` ```text "List all tables and their sizes in prod-postgres" ``` ```text "Check for missing indexes on the orders table" ``` -------------------------------- ### PromQL Example Queries Source: https://holmesgpt.dev/latest/reference/opentelemetry Example PromQL queries for monitoring tool call rates and average LLM call duration in Prometheus or Grafana. ```promql # Tool call rate by tool name rate(holmesgpt_tool_call_count_total[5m]) # Average LLM call duration rate(gen_ai_client_operation_duration_sum[5m]) / rate(gen_ai_client_operation_duration_count[5m]) ``` -------------------------------- ### MongoDB toolset example queries Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/mongodb?q= Example natural-language queries for the MongoDB toolset. Each query maps to one of the available tools (e.g., list_collections, find, aggregate, current_op, server_status). Use these as prompts to exercise the corresponding tool. ```text "List all collections in the orders database and show their schemas" ``` ```text "Find the 10 most recent error documents in the events collection" ``` ```text "How many orders per status are there in the last 24 hours?" ``` ```text "Are there any slow queries running on the MongoDB server?" ``` ```text "Check the MongoDB server health — connections, memory, and lock statistics" ``` -------------------------------- ### Dynatrace DQL Example Queries Source: https://holmesgpt.dev/latest/reference/opentelemetry Example DQL queries for analyzing tool call duration, token usage, investigation counts, and tool performance in Dynatrace. ```sql # Tool call duration by tool name timeseries avg(holmesgpt.tool.call.duration, default:0), by: {holmesgpt_tool_name} # Token usage by model timeseries sum(gen_ai.client.token.usage, default:0), by: {gen_ai_request_model, gen_ai_token_type} # Investigation count over time timeseries sum(holmesgpt.investigation.count, default:0) # Slowest tools (p95) timeseries percentile(holmesgpt.tool.call.duration, 95), by: {holmesgpt_tool_name} ``` -------------------------------- ### SQLite query: suggest indexes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlite Example natural-language query to get index suggestions for a table. ```text "Suggest indexes for the orders table" ``` -------------------------------- ### Example MySQL usage: find largest tables Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-mysql Literal prompt to get 10 largest tables. ```text "What are the 10 largest tables?" ``` -------------------------------- ### Prompt: Show table structure and suggest indexes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-mysql?q= Example prompt to show table structure for products and suggest indexes. ```text "Show table structure for products and suggest indexes" ``` -------------------------------- ### Piped command example: kubectl get pods | grep error | head -10 Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/bash?q= Each segment of a piped command is validated independently. For this command to run, `kubectl get`, `grep`, and `head` must all be in the allow list. ```bash kubectl get pods | grep error | head -10 ``` -------------------------------- ### holmes ask examples for Kubernetes remediation Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp?q= Four example invocations of the `holmes ask` CLI for common Kubernetes remediation tasks: reading a file from a pod, checking DNS/endpoint reachability, restarting a deployment, and investigating/fixing crashlooping pods. Each command is a single-line shell invocation. ```bash holmes ask "Read /app/config.yaml from the checkout-api pod and tell me what database host it points to" ``` ```bash holmes ask "From inside the production cluster, check whether the payments service DNS resolves and the endpoint is reachable" ``` ```bash holmes ask "Restart the payment-service deployment in the production namespace" ``` ```bash holmes ask "The checkout-api pods are crashlooping - investigate and fix" ``` -------------------------------- ### Example Usage of Datadog Toolset Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/datadog Use the holmes ask command to query various Datadog resources. ```bash # List all monitors holmes ask "show me all Datadog monitors" # Get dashboard details holmes ask "retrieve my application dashboard from Datadog" # Check SLO status holmes ask "what's the current status of our API availability SLO?" # Search incidents holmes ask "find recent incidents in Datadog" # Get synthetic test results holmes ask "show me the latest synthetic test results for our homepage" ``` -------------------------------- ### Endpoint whitelist hosts patterns example Source: https://holmesgpt.dev/latest/data-sources/api-toolsets?q= Example of the hosts list in the endpoint whitelist. Shows bare hostnames, wildcards, scheme+port pins, and the :* wildcard for any port. Malformed entries fail at config-load time. ```yaml hosts: - "api.example.com" # any scheme, any port - "*.internal.example.com" # any subdomain, any scheme, any port - "https://jenkins.example.com:8080" # https on 8080 only - "https://*.tools.example.com:*" # https on any port, any subdomain ``` -------------------------------- ### Analyze Execution Plan Use Case Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlserver?q= Example prompt for analyzing a query execution plan. ```text "Analyze execution plan for: SELECT * FROM Orders WHERE CustomerId = 123" ``` -------------------------------- ### Configure and Use AWS Bedrock Source: https://holmesgpt.dev/latest/installation/cli-installation Set AWS credentials, install boto3, create a test pod, and run a query. ```bash export AWS_ACCESS_KEY_ID="your-access-key" export AWS_SECRET_ACCESS_KEY="your-secret-key" export AWS_DEFAULT_REGION="your-region" ``` ```bash pip install "boto3>=1.28.57" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0" # Or use another model holmes ask "what is wrong with the user-profile-import pod?" --model="bedrock/" ``` -------------------------------- ### Custom toolsets.yaml with prometheus/metrics Source: https://holmesgpt.dev/latest/development/evaluations/adding-evals Example toolsets.yaml file that enables prometheus/metrics with a custom prometheus_url and disables grafana/dashboards. This is the preferred way to configure toolsets for complex setups. ```yaml toolsets: prometheus/metrics: enabled: true config: prometheus_url: "http://custom-prometheus:9090" grafana/dashboards: enabled: false # Disable specific toolsets ``` -------------------------------- ### Image Analysis with URL Source: https://holmesgpt.dev/latest/reference/http-api Example request providing an image via a direct URL. ```bash curl -X POST http:///api/chat \ -H "Content-Type: application/json" \ -d '{ "ask": "What is in this image?", "model": "anthropic/claude-sonnet-4-5-20250929", "images": [ "https://example.com/screenshot.png" ] }' ``` -------------------------------- ### Quick start with Anthropic Claude Source: https://holmesgpt.dev/latest/installation/cli-installation?q= Quick start with Anthropic Claude. Set the API key, create a test pod, and ask your first question using the specified model. You can change the model name to any Anthropic model. ```bash export ANTHROPIC_API_KEY="your-api-key" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash holmes ask "what is wrong with the user-profile-import pod?" --model="anthropic/claude-sonnet-4-5-20250929" ``` -------------------------------- ### Example Holmes CLI queries for Datadog general toolset Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/datadog?q= Example usage of the Datadog general toolset via the Holmes CLI. These commands ask Holmes to list monitors, retrieve dashboard details, check SLO status, search incidents, and get synthetic test results. ```bash # List all monitors holmes ask "show me all Datadog monitors" # Get dashboard details holmes ask "retrieve my application dashboard from Datadog" # Check SLO status holmes ask "what's the current status of our API availability SLO?" # Search incidents holmes ask "find recent incidents in Datadog" # Get synthetic test results holmes ask "show me the latest synthetic test results for our homepage" ``` -------------------------------- ### Run setup-azure-identity.sh for Azure RBAC Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp?q= Downloads and runs the recommended setup script for Azure RBAC. It creates a managed identity, assigns RBAC roles, configures federated credentials, and outputs the configuration values for the Helm chart. Requires the script to be executed with the specified flags. ```bash curl -O https://raw.githubusercontent.com/robusta-dev/holmes-mcp-integrations/master/servers/azure/setup-azure-identity.sh bash setup-azure-identity.sh --auth-method workload-identity \ --resource-group YOUR_RESOURCE_GROUP \ --aks-cluster YOUR_AKS_CLUSTER \ --all-subscriptions ``` -------------------------------- ### Holmes OSS Helm Chart: GCP service account setup and deployment Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/gcp?q= For Holmes OSS Helm Chart: clone the integration repo, run the setup script with your project and namespace, add the gcp MCP addon configuration to values.yaml, then deploy with helm upgrade. ```bash git clone https://github.com/robusta-dev/holmes-mcp-integrations.git cd holmes-mcp-integrations/servers/gcp ./setup-gcp-service-account.sh \ --project your-project-id \ --k8s-namespace holmes ``` ```yaml mcpAddons: gcp: enabled: true serviceAccountKey: secretName: "gcp-sa-key" # Optional: defaults when user doesn't specify. Holmes can query any project the SA has access to. config: project: "your-primary-project" region: "us-central1" gcloud: enabled: true observability: enabled: true storage: enabled: true ``` ```bash helm upgrade --install holmes robusta/holmes -f values.yaml ``` -------------------------------- ### Configure Host Patterns Source: https://holmesgpt.dev/latest/data-sources/api-toolsets Example configuration for the hosts list, demonstrating various combinations of schemes, subdomains, and ports. ```yaml hosts: - "api.example.com" # any scheme, any port - "*.internal.example.com" # any subdomain, any scheme, any port - "https://jenkins.example.com:8080" # https on 8080 only - "https://*.tools.example.com:*" # https on any port, any subdomain ``` -------------------------------- ### Show Database Size Use Case Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlserver?q= Example prompt for retrieving database size and file growth settings. ```text "Show database size and file growth settings" ``` -------------------------------- ### GET /api/model - List available models Source: https://holmesgpt.dev/latest/reference/http-api Returns a list of available AI models that can be used for chat. Example shows curl command and JSON response with model names. ```bash curl http:///api/model ``` ```json { "model_name": ["anthropic/claude-sonnet-4-5-20250929", "anthropic/claude-opus-4-5-20251101", "robusta"] } ``` -------------------------------- ### Quick Start: Configure and Ask a Question Source: https://holmesgpt.dev/latest/reference/python-sdk Sets up the HolmesGPT client with an Anthropic API key, creates an AI instance with core and CLI toolsets, and asks a question about failing pods. Requires the ANTHROPIC_API_KEY environment variable. ```python import os from holmes.config import Config from holmes.core.prompt import build_initial_ask_messages from holmes.core.tools import ToolsetTag # Create configuration config = Config( api_key=os.getenv("ANTHROPIC_API_KEY"), model="anthropic/claude-sonnet-4-5-20250929", ) # Create AI instance ai = config.create_toolcalling_llm( # Only load toolsets tagged CORE or CLI (excludes server-only CLUSTER toolsets) toolset_tag_filter=[ToolsetTag.CORE, ToolsetTag.CLI], # Auto-enable every toolset that works without explicit config (e.g. kubectl on PATH) enable_all_toolsets_possible=True, # Remaining params use defaults: # prerequisite_cache=PrerequisiteCacheMode.ENABLED — use cached health-check results # reuse_executor=False — create a fresh executor each call (fine for CLI) ) # Ask a question messages = build_initial_ask_messages( initial_user_prompt="what pods are failing in production?", file_paths=None, tool_executor=ai.tool_executor, runbooks=config.get_runbook_catalog(), system_prompt_additions=None, ) response = ai.call(messages) print(response.result) ``` -------------------------------- ### Get Confluence page at URL and summarize remediation steps Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/confluence?q= Use this example to fetch a specific Confluence page by URL and summarize its remediation steps. The confluence_request capability supports fetching pages. ```text Get the Confluence page at https://mycompany.atlassian.net/wiki/spaces/OPS/pages/12345 and summarize the remediation steps ``` -------------------------------- ### Minimal prompt behavior_controls example Source: https://holmesgpt.dev/latest/reference/http-api Demonstrates how to disable most prompt sections to reduce token usage and latency. ```bash curl -X POST http:///api/chat \ -H "Content-Type: application/json" \ -d '{ "ask": "List all pods in the default namespace", "behavior_controls": { "todowrite_instructions": false, "todowrite_reminder": false, "ai_safety": false, "style_guide": false, "general_instructions": false } }' ``` -------------------------------- ### Launch Interactive Mode Source: https://holmesgpt.dev/latest/walkthrough/interactive-mode Start the interactive session by running the ask command without arguments. ```bash holmes ask ``` -------------------------------- ### Listing Available Tools Source: https://holmesgpt.dev/latest/reference/python-sdk Creates an AI instance with core and CLI toolsets, then prints the loaded toolsets with their enabled status and the names of individual tools. ```python ai = config.create_toolcalling_llm( toolset_tag_filter=[ToolsetTag.CORE, ToolsetTag.CLI], enable_all_toolsets_possible=True, ) # List loaded toolsets and their status for toolset in ai.tool_executor.toolsets: print(f" {toolset.name} ({'enabled' if toolset.enabled else 'disabled'})") # List individual tools for tool_name in sorted(ai.tool_executor.tools_by_name.keys()): print(f" {tool_name}") ``` -------------------------------- ### Find on-call runbook in SRE space and get escalation contacts Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/confluence?q= Use this example to find the on-call runbook in the SRE space and retrieve escalation contacts. The confluence_request capability can list spaces and fetch pages. ```text Find the on-call runbook in the SRE space and tell me the escalation contacts ``` -------------------------------- ### Correct API Model Usage Source: https://holmesgpt.dev/latest/reference/http-api Example of using the modelList key name in an API request. ```bash # Use the modelList key name, not the actual model identifier curl -X POST http://localhost:8080/api/chat \ -H "Content-Type: application/json" \ -d '{"ask": "list pods", "model": "fast-model"}' ``` -------------------------------- ### Define my_toolset.yaml Source: https://holmesgpt.dev/latest/reference/python-sdk?q= Example YAML structure for defining custom tools, including descriptions, prerequisites, and shell commands. ```yaml toolsets: my-service-tools: description: "Tools for checking my custom service" prerequisites: - command: "curl --version" tools: - name: check_service_health description: "Check the health endpoint of my service" command: | curl -s "${MY_SERVICE_URL}/health" - name: get_service_metrics description: "Get Prometheus-style metrics from my service" command: | curl -s "${MY_SERVICE_URL}/metrics" | head -50 ``` -------------------------------- ### Configure and Use OpenAI Source: https://holmesgpt.dev/latest/installation/cli-installation Set the API key, create a test pod, and run a query using default or custom OpenAI models. ```bash export OPENAI_API_KEY="your-api-key" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` ```bash # Uses gpt-4.1 by default (fairly fast, decent results) holmes ask "what is wrong with the user-profile-import pod?" # Or specify a different model holmes ask "what is wrong with the user-profile-import pod?" --model="gpt-5" ``` -------------------------------- ### Customize Sentry MCP LLM instructions in Robusta Helm Chart Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/sentry-mcp?q= Customize how Holmes uses Sentry by providing your own LLM instructions in the mcpAddons.sentry.llmInstructions field. The example instructs Holmes to start by listing projects, search for relevant issues, and retrieve full stack traces before drawing conclusions. ```yaml mcpAddons: sentry: enabled: true auth: secretName: "sentry-mcp-token" llmInstructions: | Use the Sentry MCP to investigate application errors and crashes. When investigating, always start by listing projects, then search for relevant issues, and retrieve full stack traces before drawing conclusions. ``` -------------------------------- ### Build Azure API MCP from source Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp?q= Build the Azure API MCP server from source by cloning the repository, building with Go, and installing the binary to /usr/local/bin. ```bash git clone https://github.com/Azure/azure-api-mcp.git cd azure-api-mcp go build -o azure-api-mcp ./cmd/server sudo mv azure-api-mcp /usr/local/bin/ ``` -------------------------------- ### Custom Python toolset for httpbin IP lookup Source: https://holmesgpt.dev/latest/reference/python-sdk Complete example of a custom Python toolset that calls httpbin.org to get the caller's IP address. It defines a Pydantic config class, a Tool subclass implementing _invoke() and get_parameterized_one_liner(), and a Toolset subclass with a prerequisites_callable() health check. The example then passes the toolset to the SDK via additional_toolsets and runs a query. Note that _toolset must be set after super().__init__() in tool classes, and _invoke() returns StructuredToolResult with SUCCESS or ERROR status. ```python import os import requests from typing import Any, ClassVar, Dict, List, Tuple, Type from pydantic import BaseModel from holmes.config import Config from holmes.core.prompt import build_initial_ask_messages from holmes.core.tools import ( CallablePrerequisite, StructuredToolResult, StructuredToolResultStatus, Tool, ToolInvokeContext, Toolset, ToolsetTag, ) # 1. Config class class HttpBinConfig(BaseModel): api_url: str = "https://httpbin.org" # 2. Tool class - implements the actual HTTP call class GetIp(Tool): def __init__(self, toolset: "HttpBinToolset"): super().__init__( name="httpbin_get_ip", description="Get the caller's public IP address via httpbin.org", parameters={}, ) # Important: set _toolset AFTER super().__init__() (Pydantic requirement) self._toolset = toolset def _invoke(self, params: dict, context: ToolInvokeContext) -> StructuredToolResult: resp = requests.get(f"{self._toolset.get_config().api_url}/ip", timeout=10) if resp.status_code != 200: return StructuredToolResult( status=StructuredToolResultStatus.ERROR, data=f"HTTP {resp.status_code}: {resp.text}", params=params, ) return StructuredToolResult( status=StructuredToolResultStatus.SUCCESS, data=resp.json(), params=params, ) def get_parameterized_one_liner(self, params: Dict) -> str: return "httpbin: Get IP address" # 3. Toolset class - groups tools + health check class HttpBinToolset(Toolset): config_classes: ClassVar[List[Type[HttpBinConfig]]] = [HttpBinConfig] def __init__(self): super().__init__( name="httpbin", description="Tools for testing HTTP requests via httpbin.org", enabled=True, prerequisites=[CallablePrerequisite(callable=self.prerequisites_callable)], tools=[GetIp(self)], tags=[ToolsetTag.CORE], ) self._httpbin_config = HttpBinConfig() def get_config(self) -> HttpBinConfig: return self._httpbin_config def prerequisites_callable(self, config: dict[str, Any]) -> Tuple[bool, str]: try: self._httpbin_config = HttpBinConfig(**config) if config else HttpBinConfig() resp = requests.get(f"{self._httpbin_config.api_url}/get", timeout=5) if resp.ok: return True, "httpbin.org reachable" return False, f"Health check failed: HTTP {resp.status_code}" except Exception as e: return False, f"Cannot reach httpbin.org: {e}" # 4. Use it - pass the toolset via additional_toolsets config = Config( api_key=os.getenv("ANTHROPIC_API_KEY"), model="anthropic/claude-sonnet-4-5-20250929", additional_toolsets=[HttpBinToolset()], ) ai = config.create_toolcalling_llm( toolset_tag_filter=[ToolsetTag.CORE, ToolsetTag.CLI], enable_all_toolsets_possible=True, ) messages = build_initial_ask_messages( initial_user_prompt="What is my public IP address?", file_paths=None, tool_executor=ai.tool_executor, runbooks=config.get_runbook_catalog(), system_prompt_additions=None, ) response = ai.call(messages) print(response.result) ``` -------------------------------- ### Configure Azure SQL Connection Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlserver?q= Example configuration for the database toolset using the mssql+pytds driver. ```yaml toolsets: azure-sql-prod: type: database config: connection_url: "mssql+pytds://holmes_readonly:Your_Secure_Password123!@yourserver.database.windows.net:1433/mydb" llm_instructions: "Production Azure SQL database with application data" ``` -------------------------------- ### Launch Interactive Mode with Initial Question Source: https://holmesgpt.dev/latest/walkthrough/interactive-mode Start the interactive session and immediately provide an initial query for the AI to investigate. ```bash holmes ask "what pods are failing?" ``` -------------------------------- ### Query Azure cost spikes Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp Example prompt for analyzing sudden increases in Azure cloud spending. ```text "Our Azure costs increased 50% last week" ``` -------------------------------- ### Create GCP Service Account (Robusta Helm Chart) Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/gcp Clones the holmes-mcp-integrations repository and runs the setup script to create a service account with ~50 read-only IAM roles, generate a key, and create a Kubernetes secret named gcp-sa-key in the robusta namespace. ```bash git clone https://github.com/robusta-dev/holmes-mcp-integrations.git cd holmes-mcp-integrations/servers/gcp ./setup-gcp-service-account.sh \ --project your-project-id \ --k8s-namespace robusta ``` -------------------------------- ### Install HolmesGPT dependencies Source: https://holmesgpt.dev/latest/development/evaluations/running-evals Install the necessary Python dependencies for HolmesGPT using poetry. ```bash poetry install --with=dev ``` -------------------------------- ### Quick Start with HolmesGPT Python SDK Source: https://holmesgpt.dev/latest/installation/python-installation Configures the HolmesGPT SDK with an Anthropic API key, creates a tool-calling LLM with core and CLI toolsets, builds initial ask messages for a user prompt, and prints the response. ```python import os from holmes.config import Config from holmes.core.prompt import build_initial_ask_messages from holmes.core.tools import ToolsetTag config = Config( api_key=os.getenv("ANTHROPIC_API_KEY"), model="anthropic/claude-sonnet-4-5-20250929", ) ai = config.create_toolcalling_llm( toolset_tag_filter=[ToolsetTag.CORE, ToolsetTag.CLI], enable_all_toolsets_possible=True, ) messages = build_initial_ask_messages( initial_user_prompt="what pods are failing in production?", file_paths=None, tool_executor=ai.tool_executor, runbooks=config.get_runbook_catalog(), system_prompt_additions=None, ) response = ai.call(messages) print(response.result) ``` -------------------------------- ### Configure and Use Google Gemini Source: https://holmesgpt.dev/latest/installation/cli-installation Set Gemini environment variables and create a test pod. ```bash export GEMINI_API_KEY="your-gemini-api-key" export TOOL_SCHEMA_NO_PARAM_OBJECT_IF_NO_PARAMS=true export MODEL="gemini/" ``` ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml ``` -------------------------------- ### Install HolmesGPT Source: https://holmesgpt.dev/latest/installation/kubernetes-installation?q= Command to install the HolmesGPT Helm chart using the configured values.yaml file. ```bash helm install holmesgpt robusta/holmes -f values.yaml ``` -------------------------------- ### Apply the manifest and verify deployment Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/azure-mcp Apply the configured manifest to the cluster and verify that the new pod is running correctly. ```bash kubectl apply -f azure-mcp-additional-instance.yaml # verify it comes up (use the name/namespace you chose in Step 2) kubectl get pods -n monitoring -l app=azure-prod kubectl logs -n monitoring -l app=azure-prod ``` -------------------------------- ### Install HolmesGPT dev dependencies Source: https://holmesgpt.dev/latest/development/evaluations/adding-evals?q= Install the Python development dependencies required for running evals. ```bash poetry install --with=dev ``` -------------------------------- ### Common Use Case: Show Tables and Row Counts Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-sqlite?q= Example query to show all tables and their row counts. ```sql "Show all tables and their row counts" ``` -------------------------------- ### Test ArgoCD Toolset Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/argocd Sample question to verify the ArgoCD toolset works. ```bash holmes ask "Which ArgoCD applications are failing and why?" ``` -------------------------------- ### Example usage: analyze slow query Source: https://holmesgpt.dev/latest/data-sources/builtin-toolsets/database-postgresql Example free-text prompt for HolmesGPT to investigate a slow query. ```text "Why is this query slow: SELECT * FROM orders WHERE customer_id = 12345" ``` -------------------------------- ### Install or Upgrade Helm Release Source: https://holmesgpt.dev/latest/operator/development Commands to install, upgrade, and verify the operator deployment using the local configuration. ```bash # Install new release helm install holmesgpt helm/holmes -f values-dev.yaml # Upgrade existing release helm upgrade holmesgpt helm/holmes -f values-dev.yaml # Verify deployment kubectl get pods -l app.kubernetes.io/name=holmes-operator kubectl logs -l app.kubernetes.io/name=holmes-operator --tail=50 ``` -------------------------------- ### Install Unreleased Version from GitHub Source: https://holmesgpt.dev/latest/installation/python-installation Installs an unreleased version of the HolmesGPT package directly from the master branch on GitHub. ```bash pip install "https://github.com/HolmesGPT/holmesgpt/archive/refs/heads/master.zip" ``` -------------------------------- ### Install HolmesGPT Python Package Source: https://holmesgpt.dev/latest/installation/python-installation Installs the latest stable version of the HolmesGPT Python package using pip. ```bash pip install holmesgpt # Installs latest stable version ```