### Configure kubectl for GKE Cluster Source: https://help.doit.com/docs/kubernetes/kubernetes-lens/get-started Commands to configure the kubectl command-line tool to communicate with a Google Kubernetes Engine (GKE) cluster. Requires gcloud CLI installed and project set. ```bash gcloud config set project ``` ```bash gcloud container clusters get-credentials --zone --project ``` -------------------------------- ### Ask Ava About Multicloud Costs (CLI Example) Source: https://help.doit.com/docs/general/ava/using-ava This example shows a prompt you can use to ask Ava about your total multicloud cost over a specific period. Ava will then generate a report based on your query. ```cli What's been my total multicloud cost in the last 3 months? ``` -------------------------------- ### Configure kubectl for EKS Cluster Source: https://help.doit.com/docs/kubernetes/kubernetes-lens/get-started Command to configure the kubectl command-line tool to communicate with an Amazon EKS cluster. Requires AWS CLI installed and authenticated. ```bash aws eks update-kubeconfig --name --region ``` -------------------------------- ### Example employees.csv data Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-ingest-data This CSV file contains example data for employee operational costs, including Employee ID, Name, Territory, Team, Month, and Cost. It's used to demonstrate how to structure data for import. ```csv Employee ID,Employee Name,Territory,Team,Month,Cost E1001,John Doe,AMER,Exec,2024-03-01,$75000 E1001,John Doe,AMER,Exec,2024-04-01,$75000 E1001,John Doe,AMER,Exec,2024-05-01,$75000 E1002,Jane Smith,AMER,Legal and Finance,2024-03-01,$60000 E1002,Jane Smith,AMER,Legal and Finance,2024-04-01,$60000 E1002,Jane Smith,AMER,Legal and Finance,2024-05-01,$60000 E1003,Emily Davis,AMER,R&D,2024-03-01,$116000 E1003,Emily Davis,AMER,R&D,2024-04-01,$116000 E1003,Emily Davis,AMER,R&D,2024-05-01,$116000 E1004,William Brown,AMER,HR and Support,2024-03-01,$56000 E1004,William Brown,AMER,HR and Support,2024-04-01,$56000 E1004,William Brown,AMER,HR and Support,2024-05-01,$56000 E1005,Xiao Ming,EMEA,R&D,2024-03-01,$61000 E1005,Xiao Ming,EMEA,R&D,2024-04-01,$61000 E1005,Xiao Ming,EMEA,R&D,2024-05-01,$61000 E1006,Ahmed Khan,APAC,R&D,2024-03-01,$58000 E1006,Ahmed Khan,APAC,R&D,2024-04-01,$58000 E1006,Ahmed Khan,APAC,R&D,2024-05-01,$58000 ``` -------------------------------- ### Example netsuite.csv data Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-ingest-data This CSV file provides sample data for third-party charges from NetSuite, including Territory, Month, and Cost. It serves as an example for data ingestion. ```csv Territory,Month,Cost AMER,2024-03-01,$70000 AMER,2024-04-01,$70000 AMER,2024-05-01,$70000 EMEA,2024-03-01,$7500 EMEA,2024-04-01,$7500 EMEA,2024-05-01,$7500 APAC,2024-03-01,$3700 APAC,2024-04-01,$3700 APAC,2024-05-01,$3700 ``` -------------------------------- ### Example CSV from Third-Party Provider Source: https://help.doit.com/docs/integrations/datahub/import-data/upload-csv A sample CSV file formatted as typically received from a third-party provider, with simple headers and data. ```csv Month,Territory,Cost 2024-03-01,AMER,$70000 2024-04-01,AMER,$70000 2024-05-01,AMER,$70000 2024-03-01,EMEA,$7500 2024-04-01,EMEA,$7500 2024-05-01,EMEA,$7500 2024-03-01,APAC,$3700 2024-04-01,APAC,$3700 2024-05-01,APAC,$3700 ``` -------------------------------- ### Helm Deployment Commands for EKS Lens Source: https://help.doit.com/docs/kubernetes/eks-lens/connect-eks-clusters Installs and configures the EKS Lens Helm chart, including adding the repository, templating, and installing/upgrading the release. Requires Helm to be installed. ```shell helm repo add helm template helm upgrade --install ``` -------------------------------- ### Example Source CSV for User Interactions Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-unit-economics This CSV file contains raw product usage data showing user interactions per team per month. It serves as the initial data source for analysis. ```csv Month,Team,User Interactions 2024-03-01,platform,"1,042" 2024-04-01,platform,979 2024-05-01,platform,"1,103" 2024-03-01,adoption,575 2024-04-01,adoption,671 2024-05-01,adoption,722 2024-03-01,data,"2,206" 2024-04-01,data,"2,649" 2024-05-01,data,"2,840" 2024-03-01,growth,365 2024-04-01,growth,449 2024-05-01,growth,512 2024-03-01,qa,31 2024-04-01,qa,30 2024-05-01,qa,31 ``` -------------------------------- ### Clone DoiT Terraform Repository Source: https://help.doit.com/docs/kubernetes/eks-lens/connect-eks-clusters Clones the necessary Terraform repository from GitHub for setting up EKS Lens. It requires Git to be installed. ```shell git clone https://github.com/doitintl/terraform-eks-lens.git eks-lens-ACCOUNT-REGION cd eks-lens-ACCOUNT-REGION ``` -------------------------------- ### Example CSV Ready for DataHub Ingestion Source: https://help.doit.com/docs/integrations/datahub/import-data/upload-csv A sample CSV file prepared according to DataHub's conventions, including the required 'usage_date', dimension keys (e.g., 'label.territory'), and metric types (e.g., 'metric.cost'). Timestamps are in RFC3339 format. ```csv usage_date,label.territory,metric.cost 2024-03-01T00:00:00Z,AMER,70000 2024-04-01T00:00:00Z,AMER,70000 2024-05-01T00:00:00Z,AMER,70000 2024-03-01T00:00:00Z,EMEA,7500 2024-04-01T00:00:00Z,EMEA,7500 2024-05-01T00:00:00Z,EMEA,7500 2024-03-01T00:00:00Z,APAC,3700 2024-04-01T00:00:00Z,APAC,3700 2024-05-01T00:00:00Z,APAC,3700 ``` -------------------------------- ### Define Ingress Policy for BigQuery API Access Source: https://help.doit.com/docs/dashboards/bigquery-lens/set-up-bigquery-lens This example demonstrates how to configure an ingress policy within VPC Service Controls to allow specific identities and projects access to the BigQuery API. It details the source identities and target projects. ```bash gcloud access-context-manager perimeters ingress-policies create \ --perimeter=doit-perimeter \ --identity=serviceAccount:service-org-ORG_ID@gcp-sa-logging.iam.gserviceaccount.com \ --projects=PROJECT_ID \ --api-list=bigquery.googleapis.com \ --method-list=google.cloud.bigquery.v2.Methods \ --policy-name=accessPolicies/POLICY_ID ``` -------------------------------- ### Auto-labeling Resources with Iris 3 Source: https://help.doit.com/docs/cloud-analytics/tutorials/gcp-tags-and-labels Illustrates an example of how Iris 3, an open-source project, automates the labeling of Google Cloud resources like Compute Engine instances. ```text Example labels for a Compute Engine instance: `iris_name:nginx` `iris_region:us-central1` `iris_zone:us-central1-a` ``` -------------------------------- ### AWS CLI Command for CloudFormation Stack Source: https://help.doit.com/docs/cloud-analytics/cloud-diagrams/create This snippet shows the AWS CLI command to create a CloudFormation stack for setting up cloud diagrams. It requires the AWS CLI to be installed and configured. ```shell aws cloudformation create-stack --stack-name DoiT-Cloud-Diagrams --template-url --capabilities CAPABILITY_IAM ``` -------------------------------- ### Terraform Deployment Commands Source: https://help.doit.com/docs/kubernetes/eks-lens/connect-eks-clusters Executes the standard Terraform workflow to initialize, plan, and apply infrastructure changes. Assumes Terraform is installed and configured. ```shell terraform init terraform plan terraform apply ``` -------------------------------- ### DoiT Console SAML Configuration from Okta Source: https://help.doit.com/docs/user-management/single-sign-on/sso-okta Copy SAML configuration details from Okta's 'View SAML setup instructions' into the DoiT console's SAML settings. This includes the Identity Provider Issuer (Entity ID), Identity Provider Single Sign-On URL (SSO URL), and the x.509 Certificate. ```text Entity ID: SSO URL: Certificate: ``` -------------------------------- ### Example Tag Usage in DoiT Cloud Analytics Report Source: https://help.doit.com/docs/cloud-analytics/tutorials/gcp-tags-and-labels Demonstrates how to use a tag key with standard dimensions like Service, Project/Account ID, and SKU to analyze costs in a DoiT Cloud Analytics report. ```text Tag key: `vpc1-tag` Dimensions: **Service**, **Project/Account ID**, **SKU** ``` -------------------------------- ### Grafana Infinity Data Source Query Source: https://help.doit.com/docs/cloud-analytics/tutorials/grafana Example UQL query to process DoiT API JSON data for visualization in Grafana using the Infinity data source plugin. ```APIDOC ## Grafana Panel Configuration with Infinity Data Source ### Description Configures a Grafana panel to query DoiT Reports API data using the Infinity data source, processing the JSON response with UQL for visualization. ### Method GET ### Endpoint `https://api.doit.com/analytics/v1/reports/{id}` ### Parameters #### Query Parameters - **Type**: `JSON` - **Parser**: `UQL` - **Source**: `URL` - **Format**: `Time Series` #### HTTP Headers - **Key**: `Authorization` - **Value**: `$doit_api_key` (Grafana variable for your DoiT API key) ### Request Example (in Grafana Panel Settings) URL: `https://api.doit.com/analytics/v1/reports/your-report-id` Method: `GET` Headers: `Authorization: Bearer $doit_api_key` ### UQL Query ``` parse-json | jsonata "$map( $distinct($map($.result.rows, function($r) { $r[-1] })), function ($ts) { $merge([ { 'timestamp': $ts }, $merge($map($filter($.result.rows, function($r) { $r[-1] = $ts }), function($r) { $map( [ $merge($map($r, function($v, $i) { { $.result.schema[$i].name: $v } })) ], function($o) { { $o.project_id: $o.cost } })[0] })) ]) })" | extend "timestamp"=unixtime_seconds_todatetime("timestamp") ``` ### Response #### Success Response (200) Grafana will display the visualized data based on the UQL query and selected visualization type (e.g., bar chart). #### Response Example (processed for Grafana) This is an example of how the data might be structured after processing by the UQL query for visualization: ```json [ { "timestamp": "2025-04-01T00:00:00Z", "123123123123": 99.10132309182815 }, { "timestamp": "2025-02-01T00:00:00Z", "456456456": 620.7188140879603 } ] ``` ``` -------------------------------- ### Okta SAML 2.0 Setup in DoiT Console Source: https://help.doit.com/docs/user-management/single-sign-on/sso-okta Configure SAML settings for the DoiT console application in Okta. This includes specifying the Single Sign-On URL, Audience URI, Name ID format, and Application username. The Name ID format and Application username must be set to 'EmailAddress' and 'Email' respectively. ```text Single sign-on URL: https://console.doit.com/__/auth/handler Audience URI (SP Entity ID): Name ID format: EmailAddress Application username: Email ``` -------------------------------- ### BigQuery Lens Permissions (Cost Optimization) Source: https://help.doit.com/docs/vendor-information/security-and-data-access-policy Permissions for BigQuery Lens to get cost optimization recommendations by accessing BigQuery environment metadata. Does not grant access to actual data. ```plaintext bigquery.datasets.create bigquery.datasets.get bigquery.tables.get bigquery.tables.list bigquery.jobs.get bigquery.jobs.list bigquery.jobs.listAll bigquery.jobs.create bigquery.routines.list bigquery.routines.get logging.sinks.create logging.sinks.get ``` -------------------------------- ### Grant Organization Access with gcloud Source: https://help.doit.com/docs/google-cloud/transfer-google-cloud-projects/bulk-transfer-using-wizard This command grants the specified service account the 'Billing Account Administrator' role at the organization level. Ensure the service account email and organization ID are correctly substituted. ```bash gcloud alpha billing projects link --billing-account= --project= ``` -------------------------------- ### Condition Node Example in CloudFlow Source: https://help.doit.com/docs/operate/cloudflow/activities Illustrates a condition node in a CloudFlow used for branching logic. It checks if an Amazon EC2 instance has stopped, taking input from a preceding action node and evaluating a specific field. ```CloudFlow Condition node: Data source: [Previous Action Node] Field: [Instance State Field] Condition check: InstanceState == "stopped" ``` -------------------------------- ### Create AWS IAM Role - Manual JSON Policy Source: https://help.doit.com/docs/amazon-web-services/link-account Example of a JSON policy to be used when manually creating an IAM role in AWS for DoiT integration. This policy grants necessary permissions for core features. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:ListAccountAliases", "iam:ListUsers", "iam:ListRoles", "iam:GetAccountSummary", "iam:ListPolicies", "iam:GetPolicy", "iam:GetPolicyVersion" ], "Resource": "*" } ] } ``` -------------------------------- ### Filter Node Example in CloudFlow Source: https://help.doit.com/docs/operate/cloudflow/activities Demonstrates how a filter node in a CloudFlow processes data. It uses the output of a previous node ('2. DescribeInstances') and applies a filter criterion ('Reservations.Groups.GroupsName == "my-group"') to select specific instance IDs ('Reservations.Instances.InstanceId'). ```CloudFlow Filter node: Data source: 2. DescribeInstances Filter criteria: Reservations.Groups.GroupsName == "my-group" Output: Reservations.Instances.InstanceId ``` -------------------------------- ### JSON Payload Example for DataHub API Source: https://help.doit.com/docs/integrations/datahub/import-data/send-data-via-api This JSON payload represents a sample event to be sent to the DataHub API for creating or updating datasets. It includes essential fields like provider, a unique event ID, dimensions with keys and values, a timestamp, and metrics. ```json { "events": [ { "provider": "Product usage", "id": "beb21d99-a8c9-4dc0-8a69-5d684cc41e6c", "dimensions": [ { "key": "Team", "type": "label", "value": "platform" } ], "time": "2024-03-01T23:00:00Z", "metrics": [ { "value": 1042, "type": "User Interactions" } ] } ] } ``` -------------------------------- ### Example DoiT Reports API Response Schema and Data Source: https://help.doit.com/docs/cloud-analytics/tutorials/grafana Illustrates the structure of the JSON response from the DoiT Reports API, detailing the 'schema' for column metadata and 'rows' for the actual data. This is crucial for understanding how to query and process the information. ```json { "result": { "schema": [ { "name": "project_id", "type": "string" }, { "name": "year", "type": "string" }, { "name": "month", "type": "string" }, { "name": "cost", "type": "float" }, { "name": "timestamp", "type": "timestamp" } ], "rows": [ [ "123123123123", "2025", "04", 99.10132309182815, 1743465600 ], [ "456456456", "2025", "02", 620.7188140879603, 1738368000 ], ... ] } } ``` -------------------------------- ### AWS Error Message: Launch Configuration Not Supported Source: https://help.doit.com/docs/optimize/spot-scaling/faq Indicates that PerfectScale for Spot capabilities cannot be used with AWS launch configurations. AWS no longer supports launch configurations, and users should migrate to launch templates. ```text > Incompatible launch template: launch configuration is not supported ``` -------------------------------- ### Configure VPC Service Controls Access Level for DoiT Console Source: https://help.doit.com/docs/dashboards/bigquery-lens/set-up-bigquery-lens This section outlines the steps to create a basic access level within VPC Service Controls to manage context-aware access for the DoiT Console. It specifies the necessary attributes and network configurations. ```bash gcloud access-context-manager perimeters create doit-perimeter \ --title="DoiT Perimeter" \ --resources=projects/PROJECT_ID \ --restricted-services=bigquery.googleapis.com \ --enable-vpc-service-perimeter \ --policy-name=accessPolicies/POLICY_ID ``` -------------------------------- ### Generate gcloud Commands for Project Connection Source: https://help.doit.com/docs/google-cloud/connect-google-cloud-service-account This section details the generation of `gcloud` commands for connecting a Google Cloud project. The commands facilitate setting up access control by creating a service account, enabling APIs, and defining custom IAM roles and policies at the project level. ```bash # Example gcloud commands for project connection (conceptual) gcloud config set project gcloud iam service-accounts create doit_service_account --display-name="DoiT Service Account" gcloud services enable cloudresourcemanager.googleapis.com gcloud services enable serviceusage.googleapis.com gcloud iam roles create doit_cmp_role --project= --file=role-definition.yaml gcloud projects add-iam-policy-binding --member=serviceAccount:doit_service_account@.iam.gserviceaccount.com --role=projects//roles/doit_cmp_role gcloud iam service-accounts add-iam-policy-binding doit_service_account@.iam.gserviceaccount.com --member=serviceAccount:doit_caller@.iam.gserviceaccount.com --role=roles/iam.serviceAccountTokenCreator --project= ``` -------------------------------- ### Example Fee Calculation (EGP) Source: https://help.doit.com/docs/billing/invoices-and-payments/credit-card-fees An example demonstrating the calculation of the credit card fee for an invoice paid in Egyptian Pounds (EGP), which has a higher processing fee of 4.9%. It shows the fee calculation before and after tax. ```text Credit card fee (SKU PMCCFEE): 11.90/(1 - 0.029) - 11.90 = 0.36 Your credit card fee with tax: 0.36 + 0.07 = 0.43 Final invoice total amount with credit card fees: 11.90 + 0.43 = 12.33 ``` -------------------------------- ### Okta Bookmark App URL for DoiT Console Source: https://help.doit.com/docs/user-management/single-sign-on/sso-okta Configure the Bookmark App integration in Okta with the provided Bookmark App URL from the DoiT console. This URL is used to simulate an IdP-initiated flow and display the DoiT application in the Okta app catalog. ```text URL: https://console.doit.com/sign-in?idp_tenant_id={tenantID} ``` -------------------------------- ### Get Report Results API Source: https://help.doit.com/docs/cloud-analytics/tutorials/grafana Retrieve data from a specific DoiT report. This is useful for understanding the schema and content of your reports before integrating with Grafana. ```APIDOC ## GET /analytics/v1/reports/{id} ### Description Retrieves the results of a specific report identified by its unique ID. ### Method GET ### Endpoint `/analytics/v1/reports/{id}` ### Parameters #### Path Parameters - **id** (string) - Required - The unique identifier for the report. #### Request Body None ### Request Example ``` GET /analytics/v1/reports/your-report-id Authorization: Bearer YOUR_DOIT_API_KEY ``` ### Response #### Success Response (200) - **result** (object) - Contains the schema and rows of the report data. - **schema** (array) - An array of objects, each describing a column (name, type). - **rows** (array) - An array of arrays, where each inner array represents a row of data. #### Response Example ```json { "result": { "schema": [ { "name": "project_id", "type": "string" }, { "name": "year", "type": "string" }, { "name": "month", "type": "string" }, { "name": "cost", "type": "float" }, { "name": "timestamp", "type": "timestamp" } ], "rows": [ [ "123123123123", "2025", "04", 99.10132309182815, 1743465600 ], [ "456456456", "2025", "02", 620.7188140879603, 1738368000 ] ] } } ``` ``` -------------------------------- ### AWS Error Message: Incompatible Launch Template (Spot Instances) Source: https://help.doit.com/docs/optimize/spot-scaling/faq This AWS error occurs when attempting to create a mixed instances group with a launch template that already specifies a request for Spot Instances. The solution involves using a launch template without this specific setting. ```text > Incompatible launch template: You cannot use a launch template that is set to request Spot Instances (`InstanceMarketOptions`) when you configure an Auto Scaling group with a mixed instances policy. Add a different launch template to the group and try again. ``` -------------------------------- ### Google Cloud Read-Only Permissions (Organization/Project) Source: https://help.doit.com/docs/vendor-information/security-and-data-access-policy Minimum read-only permissions for DoiT to get information about Google Cloud resource hierarchy and correlate with billing. ```plaintext resourcemanager.projects.get compute.addresses.list compute.disks.get compute.disks.list compute.images.get compute.images.list compute.instances.get compute.instances.list compute.projects.get compute.regions.get compute.regions.list compute.snapshots.get compute.snapshots.list compute.zones.get compute.zones.list compute.commitments.get compute.commitments.list ``` -------------------------------- ### Configure OpenTelemetry Collector Environment Variables in Terraform Source: https://help.doit.com/docs/kubernetes/eks-lens/connect-eks-clusters This snippet shows how to set environment variables for the OpenTelemetry Collector in Terraform by utilizing the `otel_env` variable. It includes examples for setting the GOMEMLIMIT. ```Terraform # If you need to set environment variables for the OpenTelemetry Collector, you can do so by setting the `otel_env` variable: # otel_env = { # "GOMEMLIMIT" = "2750MiB" # set the memory limit for the OpenTelemetry Collector # } # We recommend to read the OpenTelemetry Collector documentation to understand the memory limiter processor configuration: https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md#best-practices # If you want to customize the memory limiter processor for the OpenTelemetry Collector, you can do so by setting the `otel_memory_limiter` variable: # otel_memory_limiter = { # check_interval = "1s" # limit_percentage = 70 # spike_limit_percentage = 30 # } # If you want to customize the resources for the OpenTelemetry Collector container, you can do so by setting the `otel_resources` variable: # otel_resources = { # requests = { # cpu = "100m" # memory = "256Mi" # } # limits = { # cpu = "100m" # memory = "256Mi" # } # } ``` -------------------------------- ### Set Google Cloud project Source: https://help.doit.com/docs/general/connect-cloud/console-gcp This command configures the Google Cloud SDK to use a specific project as the default for subsequent operations. Replace $PROJECT_ID with your actual project ID. ```bash gcloud config set project $PROJECT_ID ``` -------------------------------- ### PerfectScale for Spot Auto-Enrollment Tags Source: https://help.doit.com/docs/optimize/spot-scaling/faq Tags used for PerfectScale for Spot's auto-enrollment feature, which leverages AWS ASG Tags to enable infrastructure as code for managing cloud resources. These tags are independent of the IaC tool used. ```text doitintl:spotscaling:auto-enroll doitintl:fallback-to-on-demand ``` -------------------------------- ### Create legacy CUR export settings in AWS Source: https://help.doit.com/docs/general/connect-cloud/console-aws Configuration details for creating a legacy Cost and Usage Report (CUR) export in AWS. This includes specifying the export type, additional content, data granularity, and compression. ```text Export type : Legacy CUR export Additional content : Include resource IDs, Split cost allocation data Report data time granularity : Hourly Compression type : Parquet (recommended) or GZIP ``` -------------------------------- ### DoiT Real-time Anomalies S3 Bucket Permissions Source: https://help.doit.com/docs/amazon-web-services/link-account Grants permissions for a cross-account role to list the CloudTrail bucket and retrieve its notification configuration, as well as get objects from the bucket for real-time anomaly processing. ```json { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetBucketNotification" ], "Resource": "arn:aws:s3:::${CloudTrailBucketName}", "Effect": "Allow", "Sid": "AllowAccessToCloudTrailS3BucketListObjects" }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::${CloudTrailBucketName}/*", "Sid": "AllowAccessToCloudTrailS3BucketGetObjects" } ] } ``` -------------------------------- ### Grant Temporary Access to Google Cloud Source: https://help.doit.com/docs/services/consulting-support/create-support-request This code snippet provides instructions for users to grant temporary read-only access to their Google Cloud environment to expedite support requests. It involves copying and running a provided code block within the Google Cloud console. ```bash # Example code block to be provided by DoiT console for granting access # This is a placeholder and would be replaced by actual gcloud commands echo "Please run the provided gcloud commands to grant temporary access." ``` -------------------------------- ### Manual Labeling of Cloud Storage Buckets Source: https://help.doit.com/docs/cloud-analytics/tutorials/gcp-tags-and-labels Provides step-by-step instructions for manually labeling Cloud Storage buckets in the Google Cloud console and how to use these labels in DoiT Cloud Analytics. ```text Labeling a bucket: 1. Go to Cloud Storage Buckets page. 2. Select **Edit labels** for the target bucket. 3. Add label: key=`bucket-name`, value=`bucket-demo` 4. Save changes. ``` ```text Using labels in DoiT Cloud Analytics: 1. Create a new report. 2. Use **Group by** option: select _Service_ (filter to `Cloud Storage (GCS)`), then select `bucket-name` from **Labels**. ``` -------------------------------- ### Generate gcloud Commands for Organization Connection Source: https://help.doit.com/docs/google-cloud/connect-google-cloud-service-account This section describes the process of generating `gcloud` commands to connect a Google Cloud organization. The commands are used to configure access control by creating a service account, enabling APIs, and setting up custom IAM roles and policies at the organization level. ```bash # Example gcloud commands for organization connection (conceptual) gcloud iam service-accounts create doit_service_account --display-name="DoiT Service Account" gcloud services enable cloudresourcemanager.googleapis.com gcloud services enable serviceusage.googleapis.com gcloud iam roles create doit_cmp_role --project= --file=role-definition.yaml gcloud organizations add-iam-policy-binding --member=serviceAccount:doit_service_account@.iam.gserviceaccount.com --role=projects//roles/doit_cmp_role gcloud iam service-accounts add-iam-policy-binding doit_service_account@.iam.gserviceaccount.com --member=serviceAccount:doit_caller@.iam.gserviceaccount.com --role=roles/iam.serviceAccountTokenCreator --project= ``` -------------------------------- ### Google Cloud Rightsizing Implementation Permissions Source: https://help.doit.com/docs/vendor-information/security-and-data-access-policy Permissions required to implement Rightsizing recommendations for Compute Engine instances. ```plaintext compute.instances.setMachineType compute.instances.stop compute.instances.start ``` -------------------------------- ### Create Google Cloud Project using gcloud CLI Source: https://help.doit.com/docs/google-cloud/create-google-cloud-project This command creates a new Google Cloud project and associates it with an organization. Ensure you have the necessary permissions to create projects within the specified organization. ```bash gcloud projects create $PROJECT_ID --organization=$ORGANIZATION_ID ``` -------------------------------- ### Deploy DoiT-CUR-Backfill-Stack Source: https://help.doit.com/docs/amazon-web-services/enable-cur Deploy the provided CloudFormation stack to create necessary AWS resources for CUR and initiate a cost allocation tag backfill. ```APIDOC ## POST /websites/help_doit/deploy-stack ### Description Deploys the DoiT-CUR-Backfill-Stack CloudFormation template to AWS. This stack sets up CUR and requests a historical data backfill. ### Method POST ### Endpoint /websites/help_doit/deploy-stack ### Parameters #### Request Body - **StackName** (string) - Required - A name for the CloudFormation stack. - **BucketName** (string) - Required - The name for the S3 bucket that will store CUR reports. ### Request Example ```json { "StackName": "my-cur-stack", "BucketName": "my-doit-cur-bucket-12345" } ``` ### Response #### Success Response (200) - **StackId** (string) - The unique identifier of the created CloudFormation stack. - **StackStatus** (string) - The current status of the CloudFormation stack. #### Response Example ```json { "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/my-cur-stack/abcdef12-3456-7890-abcd-ef1234567890", "StackStatus": "CREATE_IN_PROGRESS" } ``` ``` -------------------------------- ### CSV for DoiT DataHub Upload (Reformatted) Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-unit-economics This reformatted CSV file is prepared for upload to DoiT DataHub. It includes specific headers like `label.team` and `metric.user_interactions` for custom dimension and metric definition. ```csv usage_date;label.team;metric.user_interactions 2024-03-01T00:00:00Z;platform;1042 2024-04-01T00:00:00Z;platform;979 2024-05-01T00:00:00Z;platform;1103 2024-03-01T00:00:00Z;adoption;575 2024-04-01T00:00:00Z;adoption;671 2024-05-01T00:00:00Z;adoption;722 2024-03-01T00:00:00Z;data;2206 2024-04-01T00:00:00Z;data;2649 2024-05-01T00:00:00Z;data;2840 2024-03-01T00:00:00Z;growth;365 2024-04-01T00:00:00Z;growth;449 2024-05-01T00:00:00Z;growth;512 2024-03-01T00:00:00Z;qa;31 2024-04-01T00:00:00Z;qa;30 2024-05-01T00:00:00Z;qa;31 ``` -------------------------------- ### Import AWS Service Catalog Portfolio (us-east-1) Source: https://help.doit.com/docs/amazon-web-services/import-aws-orgs Imports an AWS Service Catalog portfolio into the us-east-1 region. Ensure you are in the correct region for a successful import. Multiple portfolio IDs are provided as options. ```bash aws servicecatalog import-portfolio --portfolio-id port-npjvbgaskjcos --provider-name "Amazon" aws servicecatalog import-portfolio --portfolio-id port-el7j5lgjtsz5i --provider-name "Amazon" aws servicecatalog import-portfolio --portfolio-id port-xb7rsjalewmas --provider-name "Amazon" ``` -------------------------------- ### Python Lambda Function to Move Flexsave Accounts to OU Source: https://help.doit.com/docs/optimize/flexsave/aws/flexsave-aws-doit-accounts This Python script, designed for AWS Lambda, identifies Flexsave accounts (named starting with 'fs') within an AWS Organization and moves them to a specified Organizational Unit (OU). It uses the Boto3 library to interact with AWS Organizations. Ensure `ORG_ROOT` and `DESTINATION_OU` are correctly set, either directly in the code or as environment variables. ```python import boto3 ORG_ROOT = "XXX" DESTINATION_OU = "XXX" def handle_flexsave(event, context): details = event["detail"] if details["eventName"] == "AcceptHandshake": if "responseElements" in details and "handshake" in details["responseElements"]: handshake = details["responseElements"]["handshake"] if handshake["action"] == "INVITE" and handshake["state"] == "ACCEPTED": account_id = next( ( resource["value"] for resource in handshake["resources"] if resource["type"] == "ACCOUNT" ), None, ) if account_id: handle_account(account_id) return print(f"can't find account in payload") return print("Not a relevant event") def handle_account(account_id): print(f"Processing event for account {account_id}") org_client = boto3.client("organizations") response = org_client.describe_account(AccountId=account_id) account_name = response["Account"]["Name"] if account_name.startswith("fs"): org_client.move_account( AccountId=account_id, SourceParentId=ORG_ROOT, DestinationParentId=DESTINATION_OU, ) else: print(f"Account {account_name} not associated with Flexsave") ``` -------------------------------- ### Google Compute Engine Cost Allocation Source: https://help.doit.com/docs/operate/allocations/preset-allocations Allocates data costs for Google Cloud Compute Engine based on SKUs, including any applicable to DoiT. This covers the compute resources provisioned. ```text Allocates data in accordance with Google Cloud Compute Engine pricing models, including SKUs by DoiT if applicable. ``` -------------------------------- ### AWS IAM Policy for Real-time Anomalies Source: https://help.doit.com/docs/governance/cloud-anomalies/real-time-anomaly-detection/aws This AWS IAM policy grants necessary permissions for DoiT's real-time anomaly detection. It includes permissions to get S3 bucket location, put S3 bucket notifications, describe EC2 images, and use AWS KMS for decryption if applicable. These permissions are crucial for the DoiT platform to monitor AWS resources and detect anomalies. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3BucketLocation", "Effect": "Allow", "Action": "s3:GetBucketLocation", "Resource": "arn:aws:s3:::" }, { "Sid": "S3BucketNotification", "Effect": "Allow", "Action": "s3:PutBucketNotification", "Resource": "arn:aws:s3:::" }, { "Sid": "EC2DescribeImages", "Effect": "Allow", "Action": "ec2:DescribeImages", "Resource": "*" } ] } ``` -------------------------------- ### Python: Ingest Employee Cost Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-ingest-data This Python snippet demonstrates how to ingest employee cost data. It requires a GCP project and calls an ingest function with an empty string argument. ```python GCP_PROJECT = "your-gc-project" if __name__ == "__main__": ingest_employee_cost("") ``` -------------------------------- ### Grant BigQuery dataset read access to DoiT service account Source: https://help.doit.com/docs/general/connect-cloud/console-gcp This command sequence grants the DoiT service account read access to a BigQuery dataset. It first retrieves the dataset's access policies, appends the DoiT service account with 'READER' role, and then updates the dataset with the new permissions. ```bash bq show --format=prettyjson $DATASET_ID | jq '.access = .access + [{"role": "READER", "userByEmail": "$DOIT_SERVICE_ACCOUNT"}]' >permissions.json ``` ```bash bq update --source permissions.json $DATASET_ID ``` -------------------------------- ### Python Requirements for API Integration Source: https://help.doit.com/docs/integrations/datahub/tutorial-datahub/tutorial-ingest-data Lists the Python packages necessary for interacting with Google Cloud Secret Manager, handling web requests with Flask and Werkzeug, and making HTTP requests. ```python google-cloud-secret-manager Flask==2.2.5 Werkzeug==2.0.1 pytz requests python-dotenv ``` -------------------------------- ### DoiT-CUR-Backfill-Stack CloudFormation Template Source: https://help.doit.com/docs/amazon-web-services/enable-cur Details of the CloudFormation template used to enable CUR and initiate backfill, including parameters, resources, and outputs. ```APIDOC ## CloudFormation Template: DoiT-CUR-Backfill-Stack ### Description This CloudFormation template creates and configures AWS resources for Cost and Usage Reports (CUR) and triggers a cost allocation tag backfill. ### Parameters #### Path Parameters - **None** #### Query Parameters - **None** #### Request Body - **BucketName** (string) - Required - Specifies the name of the Amazon S3 bucket to store the AWS Cost and Usage Report. ### Resources - **MyCURReportBucket** (AWS::S3::Bucket) - Creates an S3 bucket for CUR reports. - **CURReportBucketPolicy** (AWS::S3::BucketPolicy) - Attaches a policy to the S3 bucket for access by the billingreports service. - **CURReport** (AWS::CUR::ReportDefinition) - Defines the CUR settings (name, time unit, format, compression, S3 details, etc.). Depends on `CURReportBucketPolicy`. - **CostAllocationTagBackfillFunction** (AWS::Lambda::Function) - A Lambda function to initiate the cost allocation tag backfill process. Depends on `CURReport`. - **LambdaExecutionRole** (AWS::IAM::Role) - IAM role granting necessary permissions for the Lambda function. - **BackfillCustomResource** (Custom::CostAllocationTagBackfill) - Custom resource that triggers the Lambda function for backfill. ### Outputs - **BackfillStatus** (string) - The status of the cost allocation tag backfill process. ``` -------------------------------- ### Deploy Custom ARM Template in Azure Portal Source: https://help.doit.com/docs/microsoft-azure/console-azure Demonstrates how to deploy a custom Azure Resource Manager (ARM) template using the Azure Portal. This involves using a JSON template provided by DoiT to configure resources for billing data integration. ```json { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": {}, "variables": {}, "resources": [ // DoiT provided ARM template resources will be placed here ], "outputs": {} } ``` -------------------------------- ### Create EBS Snapshot using AWS Source: https://help.doit.com/docs/operate/cloudflow/tutorials/tutorial-insights Takes a snapshot of specified EBS volumes. Requires the VolumeId, which should be the output from a previous filtering step. Configuration includes accounts and regions for snapshot creation. ```bash aws ec2 create-snapshot --volume-id vol-0a1b2c3d4e5f67890 --description "Snapshot for gp2 to gp3 migration" --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=gp2-migration-snapshot}]" ``` -------------------------------- ### CloudFormation StackSet for DoiT Support Deployment Source: https://help.doit.com/docs/services/consulting-support/aws-access This CloudFormation template defines a StackSet to deploy the DoiT support stack to member accounts. It configures auto-deployment, specifies capabilities, sets operation preferences, and defines parameters for role execution and diagnostic tool access. ```CloudFormation AWSTemplateFormatVersion: 2010-09-09 Description: DoiT Support Role StackSet Parameters: OrganizationalUnitIds: Type: CommaDelimitedList Description: >- Provide the organizational root OU ID (Prefix like: r-) to apply to all the accounts under this Organization. Otherwise, provide a comma-separated list of OU ids (Prefix like: ou-). AllowedPattern: "^(ou-|r-)[a-z0-9-]+(?:,( )*(ou-|r-)[a-z0-9-]+)*$" Resources: StackSetMember: Type: "AWS::CloudFormation::StackSet" Properties: AutoDeployment: RetainStacksOnAccountRemoval: false Enabled: true Capabilities: - CAPABILITY_NAMED_IAM OperationPreferences: FailureTolerancePercentage: 0 RegionConcurrencyType: PARALLEL PermissionModel: SERVICE_MANAGED Parameters: - ParameterKey: AllowSupportRunbooksExecutions ParameterValue: "false" - ParameterKey: GrantDiagnosticToolAccess ParameterValue: "true" StackInstancesGroup: - DeploymentTargets: OrganizationalUnitIds: !Ref OrganizationalUnitIds Regions: - us-east-1 StackSetName: doit-support-gateway-stackset TemplateURL: https://doit-support.s3.amazonaws.com/doit-support.json ``` -------------------------------- ### Google Cloud Rightsizing Recommendations Permissions (Organization) Source: https://help.doit.com/docs/vendor-information/security-and-data-access-policy Permissions needed to provide Rightsizing Recommendations for Google Compute Engine instances across an organization. ```plaintext recommender.computeInstanceMachineTypeRecommendations.list compute.instances.list ``` -------------------------------- ### Grant Service Principal Warehouse Access (Databricks SQL) Source: https://help.doit.com/docs/databricks/connect-databricks This SQL script grants a service principal specific permissions to access catalogs, schemas, and tables within Databricks, including system catalog data for workspaces, billing, and compute. Replace `` with the actual service principal ID. These grants are essential for DoiT to access Databricks billing and usage data. ```sql GRANT USE CATALOG ON CATALOG system TO ``; GRANT USE SCHEMA ON SCHEMA system.access TO ``; GRANT SELECT ON TABLE system.access.workspaces_latest TO ``; GRANT USE SCHEMA ON SCHEMA system.billing TO ``; GRANT SELECT ON TABLE system.billing.usage TO ``; GRANT SELECT ON TABLE system.billing.list_prices TO ``; GRANT USE SCHEMA ON SCHEMA system.compute TO ``; GRANT SELECT ON TABLE system.compute.clusters TO ``; GRANT SELECT ON TABLE system.compute.node_timeline TO ``; GRANT SELECT ON TABLE system.compute.warehouses TO ``; ``` -------------------------------- ### Apply Kubernetes Deployment YAML Source: https://help.doit.com/docs/kubernetes/eks-lens/connect-eks-clusters Applies a Kubernetes deployment configuration file using kubectl to update cluster configuration. Requires kubectl and the deployment YAML file. ```shell kubectl apply -f DEPLOYMENT_YAML_FILE ``` -------------------------------- ### Grant BigQuery Data Viewer role using bq commands Source: https://help.doit.com/docs/google-cloud/import-historical-billing-data This snippet grants the BigQuery Data Viewer role to a service account for a specific dataset using the `bq` command-line tool. It requires setting the project, showing dataset information, and updating the dataset with the service account's read access. The `jq` utility is used for JSON manipulation. ```bash gcloud config set project $customer_billing_project_id bq show --format=prettyjson $DATASET_ID | \ jq '.access = .access + [{"role": "READER", "userByEmail": "$customer_service_account"}]' > permissions.json bq update --source permissions.json $DATASET_ID ```