CloudQuery AWS Source

https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/
Admin
The CloudQuery AWS Source plugin extracts information from various Amazon Web Services (AWS) and...
Tokens:11,323
Snippets:92
Trust Score:9.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Event-based Sync Start Time Format

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Example of the RFC 3339 timestamp format required for the `start_time` parameter in event-based synchronization.

```Text
2023-09-04T19:24:14Z
```

--------------------------------

### AWS Credentials File Example

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

An example of an AWS credentials file showing different profiles that can be used for authentication.

```INI
[default]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx

[user1]
aws_access_key_id=xxxx
aws_secret_access_key=xxxx

```

--------------------------------

### Configure CloudQuery AWS Source - Basic

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

A basic configuration for the CloudQuery AWS Source plugin. This example specifies the source name, path, registry, version, desired tables, and destination. It also includes commented-out optional parameters for customization.

```yaml
kind: source
spec:
  # Source spec section
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0"
  tables: ["aws_ec2_instances"]
  destinations: ["postgresql"]
  # Learn more about the configuration options at https://cql.ink/aws_source
  spec:
    # Optional parameters
    # regions: []
    # accounts: []
    # org: nil
    # concurrency: 50000
    # initialization_concurrency: 4
    # aws_debug: false
    # max_retries: 10
    # max_backoff: 30
    # custom_endpoint_url: ""
    # custom_endpoint_hostname_immutable: nil # required when custom_endpoint_url is set
    # custom_endpoint_partition_id: "" # required when custom_endpoint_url is set
    # custom_endpoint_signing_region: "" # required when custom_endpoint_url is set
    # use_paid_apis: false
    # table_options: nil
    # scheduler: shuffle # options are: dfs, round-robin, shuffle, or shuffle-queue
    # use_nested_table_rate_limiting: false 
    # enable_api_level_tracing: false 


```

--------------------------------

### Configure CloudQuery AWS Source - Organization Example

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Configuration for the CloudQuery AWS Source plugin to discover AWS Accounts via AWS Organizations. This setup enables automatic handling of new or removed accounts. It specifies the admin account, member role name, and uses '*' to include all regions.

```yaml
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0"
  tables: ['aws_s3_buckets']
  destinations: ["postgresql"]
  spec:
    aws_debug: false
    org:
      admin_account:
        local_profile: "<NAMED_PROFILE>"
      member_role_name: OrganizationAccountAccessRole
    regions:
      - '*'
    # Optional parameters
    # regions: []
    # accounts: []
    # org: nil
    # concurrency: 50000
    # initialization_concurrency: 4
    # aws_debug: false
    # max_retries: 10
    # max_backoff: 30
    # custom_endpoint_url: ""
    # custom_endpoint_hostname_immutable: nil # required when custom_endpoint_url is set
    # custom_endpoint_partition_id: "" # required when custom_endpoint_url is set
    # custom_endpoint_signing_region: "" # required when custom_endpoint_url is set
    # use_paid_apis: false
    # table_options: nil
    # scheduler: shuffle # options are: dfs, round-robin or shuffle


```

--------------------------------

### Sync CloudQuery Configuration

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Executes the CloudQuery synchronization process using a specified configuration file. This starts a long-lived process for data syncing.

```bash
cloudquery sync config.yml
```

--------------------------------

### AWS Credentials File Configuration

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Example content for the AWS credentials file, specifying default access key ID and secret access key. The CloudQuery plugin prioritizes this file for authentication.

```ini
[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

```

--------------------------------

### AWS Source Plugin: Skip Specific APIs Example

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Demonstrates how to configure the `skip_specific_apis` option in the AWS source plugin to exclude specific API actions for certain AWS services. This is useful for fine-tuning data synchronization and avoiding unnecessary API calls.

```yaml
spec:
  regions: ["us-east-1","us-east-2"]
  skip_specific_apis:
    lambda:
      GetRuntimeManagementConfig: true
      GetFunction: true
```

--------------------------------

### Get AWS Session Token with MFA

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Obtain temporary AWS credentials by using the `aws sts get-session-token` command with your IAM User's long-term security credentials and MFA token. These temporary credentials can then be exported to environment variables.

```bash
aws sts get-session-token --serial-number <YOUR_MFA_SERIAL_NUMBER> --token-code <YOUR_MFA_TOKEN_CODE> --duration-seconds 3600

```

```bash
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
export AWS_SESSION_TOKEN=<YOUR_SESSION_TOKEN>

```

--------------------------------

### CloudQuery AWS Plugin Versioning Strategy

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Explains the CloudQuery AWS plugin's versioning strategy, differentiating between major and minor version bumps based on schema, configuration, and permission changes. Recommends specific table syncing and skipping dependent tables.

```markdown
## Versioning #
Changes to schema, configurations and required user permissions are all factors that go into the versioning of the AWS plugin. Any release that requires manual changes to an existing deployment of the AWS plugin in order to retain the same functionality will be indicated by an increase to the major version. When support for additional resources is added it will result in a minor version bump. This is important to be aware of because if you are using `tables: ["*"]` to specify the set of tables to sync then in minor versions new resources that might require additional IAM permissions might result in errors being raised.
It is recommended that you specify the exact tables you wish to sync rather than using wildcard characters as well as specifying `skip_dependent_tables: true` to limit risk while upgrading versions as new resources that can take significant time to sync can be added.
#### Breaking changes #
The following examples are some of the most common examples of reasons for a major version change:
  1. Changing a primary key for a table
  2. Changing the name of a table
  3. Changing the permissions required to sync a resource

All releases contain a change log that indicates all the changes (and highlights the breaking changes). If you are ever unsure about a change that is included feel free to reach out to the CloudQuery team on Discord to find out more.
#### Preview features #
Sometimes features or tables will be released and marked as `alpha`. This indicates that future minor versions might change, break or remove functionality. This enables the CloudQuery team to release functionality prior to it being fully stable so that the community can give feedback. Once a feature is released as Generally Available then all of the above rules for semantic versioning will apply.
##### Current Preview features
The following features are currently in `Preview`:
  * All tables that are prefixed with `aws_alpha_`
  * `table_options` feature
```

--------------------------------

### Move table options docs to tables descriptions

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This refactoring moves table options documentation into the table descriptions themselves, making it easier to find and manage configuration details for each table. This was done in release v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/tables.go
```

--------------------------------

### Support EC2 Start/Stop Instances in Event-Based Sync (Go)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

Enables event-based synchronization for EC2 `StartInstances` and `StopInstances` actions. This feature allows for near real-time updates on EC2 instance state changes.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/service/ec2/instance
```

--------------------------------

### Deploy CloudFormation for Kinesis Stream

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Deploys a CloudFormation stack to set up a Kinesis Data Stream for CloudTrail events. Requires AWS CLI and a local YAML template file.

```bash
aws cloudformation deploy --template-file ./streaming-deployment.yml --stack-name <STACK-NAME> --capabilities CAPABILITY_IAM --disable-rollback --region <DESIRED-REGION>
```

--------------------------------

### Add aws_kendra_faqs table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update introduces the `aws_kendra_faqs` table, enabling the extraction of AWS Kendra frequently asked questions (FAQ) data. This was done in v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/faqs.go
```

--------------------------------

### AWS Table Options Overview

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Provides a general structure for defining table options in CloudQuery for various AWS services. It shows how to map table names to API methods and specify input objects or parameters for each.

```yaml
table_options:
  aws_accessanalyzer_analyzer_findings:
    ListFindings:
      - <AccessAnalyzer.ListFindings> # NextToken & AnalyzerArn are prohibited

  aws_accessanalyzer_analyzer_findings_v2:
    ListFindingsV2:
      - <AccessAnalyzer.ListFindingsV2> # NextToken & AnalyzerArn are prohibited

  aws_cloudtrail_events:
    LookupEvents:
      - <CloudTrail.LookupEvents> # NextToken is prohibited

  aws_cloudtrail_trails:
    DescribeTrails:
      - <CloudTrail.DescribeTrails>

  aws_cloudwatch_metrics:
    - ListMetrics: <CloudWatch.ListMetrics> # NextToken is prohibited
      GetMetricData:
        - <CloudWatch.GetMetricData>  # MaxDatapoints, NextToken and ScanBy are prohibited
      GetMetricStatistics:
        - <CloudWatch.GetMetricStatistics>  # Namespace, MetricName and Dimensions are prohibited
    
  aws_cloudwatchlogs_delivery_destinations:
    DescribeDeliveryDestinations:
      - <CloudWatchLogs.DescribeDeliveryDestinationsInput> # NextToken is prohibited

  aws_cloudwatchlogs_delivery_sources:
    DescribeDeliverySources:
      - <CloudWatchLogs.DescribeDeliverySourcesInput> # NextToken is prohibited

  aws_cloudwatchlogs_log_groups:
    - DescribeGroups: <CloudWatchLogs.DescribeLogGroupsInput> # NextToken is prohibited
      DescribeStreams:
        cloudwatchStreamLastEventTimeAfter: <time> 
    
  aws_costexplorer_cost_custom:
    GetCostAndUsage:
      - <CostExplorer.GetCostAndUsage> # NextPageToken is prohibited

  aws_ec2_images:
    DescribeImages:
      - <EC2.DescribeImages> # NextToken and ImageIds are prohibited. MaxResults should be in range [1-1000].

  aws_ec2_instances:
    DescribeInstances:
      - <EC2.DescribeInstances> # NextToken is prohibited. MaxResults should be in range [1-1000].

  aws_ec2_internet_gateways:
    DescribeInternetGateways:
      - <EC2.DescribeInternetGateways> # NextToken is prohibited. MaxResults should be in range [5-1000].

  aws_ec2_network_interfaces:
    DescribeNetworkInterfaces:
      - <EC2.DescribeNetworkInterfaces> # NextToken is prohibited. MaxResults should be in range [5-1000].

  aws_ec2_route_tables:
    DescribeRouteTables:
      - <EC2.DescribeRouteTables> # NextToken is prohibited. MaxResults should be in range [5-100].

  aws_ec2_security_groups:
    DescribeSecurityGroups:
      - <EC2.DescribeSecurityGroups> # NextToken is prohibited. MaxResults should be in range [5-1000].

  aws_ec2_subnets:
    DescribeSubnets:
      - <EC2.DescribeSubnets> # NextToken is prohibited. MaxResults should be in range [5-1000].

  aws_ec2_vpcs:
    DescribeVpcs:
      - <EC2.DescribeVpcs> # NextToken is prohibited. MaxResults should be in range [5-1000].

  aws_ecs_cluster_tasks:
    ListTasks:
      - <ECS.ListTasks> # Cluster and NextToken are prohibited. MaxResults should be in range [1-100].

  aws_guardduty_detectors:
    - ListDetectors: <GuardDuty.ListDetectors> # NextToken is prohibited
      ListFindings: <GuardDuty.ListFindings> # NextToken and DetectorID are prohibited

  aws_iam_groups:
    GetGroup:
      - <IAM.GetGroup> # Marker is prohibited. MaxItems should be in range [1-1000].

  aws_iam_policies:
    ListPolicies:
      - <IAM.ListPolicies> # Marker is prohibited. MaxItems should be in range [1-1000].

  aws_iam_roles:
    GetRole:
      - <IAM.GetRole> # RoleName is required.

  aws_iam_users:
    GetUser:
      - <IAM.GetUser> # UserName is required.

  aws_inspector_findings:
    ListFindings:
      - <Inspector.ListFindings> # NextToken is prohibited. MaxResults should be in range [1-500].

  aws_inspector2_covered_resources:
    ListCoverage:

```

--------------------------------

### Update aws-sdk-go-v2 dependencies

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This entry represents updates to the `aws-sdk-go-v2` monorepo dependency, ensuring the plugin uses the latest versions of the AWS SDK for Go. These updates were applied in multiple sub-versions of v27.19.0.

```go
go.mod
```

--------------------------------

### Add aws_kendra_access_control_configurations table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This release introduces the `aws_kendra_access_control_configurations` table, enabling the extraction of Kendra access control configuration details. This was added in v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/access_control_configurations.go
```

--------------------------------

### Add aws_kendra_query_suggestions_configs table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update introduces the `aws_kendra_query_suggestions_configs` table, enabling the extraction of AWS Kendra query suggestions configurations. This was part of v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/query_suggestions_configs.go
```

--------------------------------

### Support AWS Free Tier API

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This feature introduces support for the AWS Free Tier API, allowing CloudQuery to extract and process free tier usage data. This was a key addition in release v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/freetier/usage.go
```

--------------------------------

### Attach SQS Policy

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Applies the defined SQS queue policy using the AWS CLI. Requires the queue URL and the path to the policy JSON file.

```bash
aws sqs set-queue-attributes --queue-url <queue_url> --policy file://sqs-policy.json
```

--------------------------------

### Handle IAM Policies in AWS Event-Based Sync (Go)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

Adds support for handling AWS IAM policies within the event-based synchronization mechanism. This improves the accuracy and timeliness of IAM policy data.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/service/iam/policy
```

--------------------------------

### AWS InspectorV2 ListFindings Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for listing AWS InspectorV2 findings. Prohibits NextToken.

```yaml
aws_inspector2_findings:
  ListFindings:
    - <InspectorV2.ListFindings> # NextToken is prohibited.
```

--------------------------------

### Add aws_kendra_data_sources table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update adds the `aws_kendra_data_sources` table, allowing for the retrieval of AWS Kendra data source configurations. This was included in release v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/data_sources.go
```

--------------------------------

### Configure AWS Plugin with Skipped Tables

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This configuration demonstrates how to set up the CloudQuery AWS source plugin, specifying all tables ('*') but explicitly skipping a list of tables known to be large or unnecessary for typical use cases. This optimizes data synchronization by reducing the amount of data processed.

```yaml
kind: source
spec:
  # Source spec section
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0"
  tables: ["*"]
  skip_tables:
    - aws_cloudtrail_events
    - aws_cloudwatchlogs_log_streams
    - aws_docdb_cluster_parameter_groups
    - aws_docdb_engine_versions
    - aws_ec2_instance_types
    - aws_ec2_vpc_endpoint_services
    - aws_elasticache_engine_versions
    - aws_elasticache_parameter_groups
    - aws_elasticache_reserved_cache_nodes_offerings
    - aws_elasticache_service_updates
    - aws_elasticbeanstalk_platform_versions
    - aws_elasticsearch_versions
    - aws_emr_release_labels
    - aws_emr_supported_instance_types
    - aws_iam_group_last_accessed_details
    - aws_iam_policy_last_accessed_details
    - aws_iam_role_last_accessed_details
    - aws_iam_user_last_accessed_details
    - aws_neptune_cluster_parameter_groups
    - aws_neptune_db_parameter_groups
    - aws_rds_cluster_parameter_groups
    - aws_rds_db_parameter_groups
    - aws_rds_engine_versions
    - aws_servicequotas_quota_utilizations
    - aws_servicequotas_services
    - aws_stepfunctions_map_run_executions
    - aws_stepfunctions_map_runs
  destinations: ["postgresql"]
  spec:
    # AWS Spec section described below
```

--------------------------------

### AWS Source Plugin: Supported API Actions

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Lists the AWS services and their specific API actions that are supported for configuration within the `skip_specific_apis` feature of the AWS source plugin. This helps users identify which API calls can be targeted for skipping.

```yaml
lambda:
  GetRuntimeManagementConfig
  GetFunction
  GetFunctionCodeSigningConfig
  GetFunctionConcurrency
kms:
  DescribeKey
  GetKeyRotationStatus
  ListResourceTags
ssm:
  ListTagsForResource
glacier:
  ListTagsForVault
wafv2:
  ListResourcesForWebACL
```

--------------------------------

### AWS RDS Instance Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for describing AWS RDS instances. Also includes GetResourceMetrics with specific time and period requirements.

```yaml
aws_rds_instances:
    DescribeDBInstances: <RDS.DescribeDBInstances> # Marker is prohibited. MaxRecords should be in range [20-100].
    GetResourceMetrics: <PI.GetResourceMetrics> # NextToken, ServiceType and Identifier are prohibited. StartTime, EndTime and MetricQueries are required. MaxResults should be in range [1-25]. PeriodInSeconds should be in range [1-86400].
```

--------------------------------

### Add table_options to aws_ec2_ebs_snapshots

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This feature adds `table_options` support for the `aws_ec2_ebs_snapshots` table, enhancing its configuration capabilities within CloudQuery. This change is part of release v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/ec2/snapshot.go
```

--------------------------------

### Add aws_kendra_featured_results_sets table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_featured_results_sets` table has been added to provide access to AWS Kendra featured results set configurations. This was part of v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/featured_results_sets.go
```

--------------------------------

### AWS CloudFormation Template for CloudTrail to Kinesis

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This is the main CloudFormation template that defines all the AWS resources. It sets up a Kinesis stream, IAM roles for log forwarding, an S3 bucket for CloudTrail logs, a CloudWatch Logs group, and the CloudTrail trail. The template allows for either creating a new S3 bucket or using an existing one.

```yaml
AWSTemplateFormatVersion: 2010-09-09
Description: Configures Cloudtrail Events to be piped to a Kinesis Data stream via CloudWatch Logs.

Parameters:

  KinesisMessageDuration:
    Type: Number
    Description: Number of hours Kinesis will persist a record before it is purged.
    Default: 24
  ExistingS3BucketName:
    Type: String
    Description: Name of the S3 Bucket that CloudTrail will use to store logs.
    Default: ""

Conditions:
  CreateS3Bucket: !Equals [!Ref ExistingS3BucketName, ""]

Resources:
  # Stream that CQ will poll for changes
  CQSyncingKinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      ShardCount: 1
      RetentionPeriodHours: !Ref KinesisMessageDuration

  # IAM Role for allowing CloudWatch Log to write to Kinesis Stream
  CloudWatchLogsToKinesisRole:
    Type: AWS::IAM::Role
    Properties:
      Policies:
      - PolicyName: CloudWatchLogsToKinesisPolicy
        PolicyDocument: 
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - kinesis:PutRecord
              Resource: !GetAtt CQSyncingKinesisStream.Arn
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: logs.amazonaws.com
            Action:
              - sts:AssumeRole


  CloudTrailS3Bucket:
    Type: AWS::S3::Bucket
    Condition: CreateS3Bucket
    Properties:
      LifecycleConfiguration:
        Rules:
          - ExpirationInDays: 30
            Status: Enabled

  CloudTrailS3BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName]
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: AWSCloudTrailAclCheck
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: !Sub
                - arn:${AWS::Partition}:s3:::${Bucket}
                - { Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] }
            Condition:
              StringEquals:
                'aws:SourceAccount': !Sub ${AWS::AccountId}
          - Sid: AWSCloudTrailWrite
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub
                - arn:${AWS::Partition}:s3:::${Bucket}/*
                - { Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] }
            Condition:
              StringEquals:
                's3:x-amz-acl': bucket-owner-full-control
                'aws:SourceAccount': !Sub ${AWS::AccountId}

  CloudWatchLogsGroup:
    Type: AWS::Logs::LogGroup
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Properties:
      LogGroupName: "CloudTrailLogGroup"
      RetentionInDays: 1

  # Role for allowing CLoudTrail to write to CloudWatch Logs
  CloudWatchRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: AssumeRole
          Effect: Allow
          Principal:
            Service: 'cloudtrail.amazonaws.com'
          Action: 'sts:AssumeRole'
      Policies:
      - PolicyName: 'cloudtrail-policy'
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action: 'logs:CreateLogStream'
            Resource: !GetAtt CloudWatchLogsGroup.Arn
          - Effect: Allow
            Action: 'logs:PutLogEvents'
            Resource: !GetAtt CloudWatchLogsGroup.Arn
  CloudTrailTrail:
    Type: AWS::CloudTrail::Trail
    DependsOn:
         - CloudTrailS3BucketPolicy
    Properties:
      CloudWatchLogsLogGroupArn: !GetAtt CloudWatchLogsGroup.Arn
      CloudWatchLogsRoleArn: !GetAtt CloudWatchRole.Arn
      EventSelectors:
        - IncludeManagementEvents: True
          ReadWriteType: WriteOnly
      IncludeGlobalServiceEvents: True
      IsLogging: True
      IsMultiRegionTrail: True
      S3BucketName: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName]

  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: !Ref CloudWatchLogsGroup
      DestinationArn: !GetAtt CQSyncingKinesisStream.Arn
      RoleArn: !GetAtt CloudWatchLogsToKinesisRole.Arn
      FilterPattern: ""


Outputs:
  KinesisStreamArn:
    Description: The ARN of the Kinesis Data Stream that CloudQuery will use to listen for changes.
    Value: !GetAtt CQSyncingKinesisStream.Arn

```

--------------------------------

### Skip kms:ListResourceTags API calls

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update allows users to skip `kms:ListResourceTags` API calls, which can optimize sync performance by reducing unnecessary API requests. This was implemented in v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kms/client.go
```

--------------------------------

### Add table_options to aws_ssm_command_invocations

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update enables `table_options` for the `aws_ssm_command_invocations` table, allowing for more flexible configuration. This was introduced in release v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/ssm/command_invocation.go
```

--------------------------------

### Add aws_kendra_data_source_sync_jobs table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_data_source_sync_jobs` table has been added to facilitate the extraction of AWS Kendra data source synchronization job information. This was part of v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/data_source_sync_jobs.go
```

--------------------------------

### Create SQS Queue for S3 Notifications

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Creates an Amazon SQS queue using the AWS CLI, which will be used to receive S3 bucket notifications.

```bash
aws sqs create-queue --queue-name <REPLACE_WITH_QUEUE_NAME> bucket-notifications
```

--------------------------------

### Add aws_kendra_thesauri table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_thesauri` table has been added to provide access to AWS Kendra thesauri configurations. This was included in release v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/thesauri.go
```

--------------------------------

### Add aws_kendra_experience_entities table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_experience_entities` table has been added to retrieve information about entities within AWS Kendra experiences. This was included in release v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/experience_entities.go
```

--------------------------------

### Configure AWS CloudTrail LookupEvents with Table Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Demonstrates how to use `table_options` to specify multiple input objects for the `LookupEvents` API in CloudTrail. This is useful for making multiple API calls with different parameters, such as varying time ranges or lookup attributes.

```yaml
table_options:
  aws_cloudtrail_events:
    LookupEvents:
      - StartTime: 2023-05-01T20:20:52Z
        EndTime:   2023-05-03T20:20:52Z
        LookupAttributes:
          - AttributeKey:   EventName
            AttributeValue: RunInstances
      - StartTime: 2023-05-01T20:20:52Z
        EndTime:   2023-05-03T20:20:52Z
        LookupAttributes:
          - AttributeKey:   EventName
            AttributeValue: StartInstances
      - StartTime: 2023-05-01T20:20:52Z
        EndTime:   2023-05-03T20:20:52Z
        LookupAttributes:
          - AttributeKey:   EventName
            AttributeValue: StopInstances
```

--------------------------------

### Add identity_center_application_arn to aws_s3_access_grant_instances

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update adds the `identity_center_application_arn` column of type `utf8` to the `aws_s3_access_grant_instances` table, enhancing its data schema. This change was made in v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/s3control/access_grant_instances.go
```

--------------------------------

### AWS IAM Access Analyzer Archive Rules

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/tables

Fetches data about AWS Identity and Access Management (IAM) Access Analyzer Analyzer Archive Rules. This resource requires 'access-analyzer:ListArchiveRules' permission.

```APIDOC
## GET /aws/accessanalyzer/analyzer_archive_rules

### Description
Retrieves a list of archive rules for AWS IAM Access Analyzer.

### Method
GET

### Endpoint
/aws/accessanalyzer/analyzer_archive_rules

### Parameters
#### Query Parameters
- **analyzer_arn** (string) - Required - The ARN of the analyzer.
- **rule_name** (string) - Required - The name of the archive rule.

### Request Example
```json
{
  "analyzer_arn": "arn:aws:iam::123456789012:analyzer/my-analyzer",
  "rule_name": "my-archive-rule"
}
```

### Response
#### Success Response (200)
- **account_id** (string) - The AWS account ID.
- **analyzer_arn** (string) - The ARN of the analyzer.
- **_cq_id** (uuid) - The CloudQuery internal ID.
- **_cq_parent_id** (uuid) - The CloudQuery parent resource ID.
- **created_at** (timestamp) - The creation timestamp.
- **filter** (json) - The filter associated with the archive rule.
- **region** (string) - The AWS region.
- **rule_name** (string) - The name of the archive rule.
- **updated_at** (timestamp) - The update timestamp.

#### Response Example
```json
{
  "account_id": "123456789012",
  "analyzer_arn": "arn:aws:iam::123456789012:analyzer/my-analyzer",
  "_cq_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "_cq_parent_id": null,
  "created_at": "2023-10-27T10:00:00Z",
  "filter": {
    "equals": [
      {
        "property": "resourceType",
        "value": "AWS::S3::Bucket"
      }
    ]
  },
  "region": "us-east-1",
  "rule_name": "my-archive-rule",
  "updated_at": "2023-10-27T10:05:00Z"
}
```
```

--------------------------------

### Add aws_kendra_query_suggestions_block_lists table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_query_suggestions_block_lists` table has been added to manage and retrieve AWS Kendra query suggestions block list configurations. This change was made in v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/query_suggestions_block_lists.go
```

--------------------------------

### Add aws_kendra_indices table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This release adds the `aws_kendra_indices` table, allowing for the extraction of AWS Kendra index configurations. This was included in release v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/indices.go
```

--------------------------------

### Apply S3 Bucket Notification Configuration

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Sets the notification configuration for an S3 bucket using the AWS CLI. Requires the bucket name and the path to the notification configuration JSON file.

```bash
aws s3api put-bucket-notification-configuration --bucket <REPLACE_WITH_BUCKET_NAME> --notification-configuration file://s3-notification.json
```

--------------------------------

### Add aws_kendra_eperiences table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

The `aws_kendra_eperiences` table has been introduced to extract AWS Kendra experience configurations. This change was made in v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/experiences.go
```

--------------------------------

### Configure AWS Plugin for Specific Accounts using AssumeRole

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This snippet shows how to configure the AWS plugin to fetch data from multiple specific accounts in parallel by using AssumeRole, specifying account name, role ARN, and optional local profile and session name.

```yaml
accounts:
  - account_name: <AccountName_1>
    role_arn: <YOUR_ROLE_ARN_1>
    # Optional. Local Profile is the named profile in your shared configuration file (usually `~/.aws/config`) that you want to use for this specific account
    local_profile: <NAMED_PROFILE>
    # Optional. Specify the Role Session name
    role_session_name: ""
  - account_name: <AccountName_2>
    local_profile: provider
    # Optional. Role ARN we want to assume when accessing this account
    role_arn: <YOUR_ROLE_ARN_2>


```

--------------------------------

### CloudQuery AWS Config with S3 Notifications

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Defines the CloudQuery configuration to use S3 bucket notifications via an SQS queue for event-based synchronization. Specifies source plugin, tables, destinations, and the SQS queue URL.

```yaml
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0"
  tables:
    - aws_ec2_instances
    - aws_ec2_internet_gateways
    - aws_ec2_security_groups
    - aws_ec2_subnets
    - aws_ec2_vpcs
    - aws_ecs_cluster_tasks
    - aws_iam_groups
    - aws_iam_roles
    - aws_iam_users
    - aws_iam_policies
    - aws_rds_instances
  destinations: ["postgresql"]
  spec:
    event_based_sync:
      # account:
      #  local_profile: "<ROLE-NAME>"
      sqs_queue_url: <OUTPUT-FROM-CREATE-QUEUE-COMMAND>
```

--------------------------------

### Enable FIPS Compliance for AWS Plugin

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This snippet shows how to enable FIPS compliance for the CloudQuery AWS plugin by specifying a FIPS-enabled version in the configuration.

```yaml
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0-fips"
   ...
```

--------------------------------

### Add id_c_config column to aws_workspaces_directories

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This enhancement adds the `id_c_config` column of type `json` to the `aws_workspaces_directories` table, providing Identity Center configuration details. This was implemented in v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/workspaces/directories.go
```

--------------------------------

### Add aws_kendra_experience_entity_personas table

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This release adds the `aws_kendra_experience_entity_personas` table, providing details on personas associated with Kendra experiences. This was part of v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/kendra/experience_entity_personas.go
```

--------------------------------

### Support newly added regions in AWS services

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This update ensures the CloudQuery AWS plugin supports newly added regions across various AWS services, maintaining compatibility with the latest AWS infrastructure. This was included in v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/service/services.go
```

--------------------------------

### CloudQuery AWS Config with Kinesis

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Defines the CloudQuery configuration to use a Kinesis Data Stream for event-based synchronization. Specifies source plugin, tables, destinations, and the Kinesis stream ARN.

```yaml
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.42.0"
  tables:
    - aws_ec2_instances
    - aws_ec2_internet_gateways
    - aws_ec2_security_groups
    - aws_ec2_subnets
    - aws_ec2_vpcs
    - aws_ecs_cluster_tasks
    - aws_iam_groups
    - aws_iam_roles
    - aws_iam_users
    - aws_iam_policies
    - aws_rds_instances
  destinations: ["postgresql"]
  spec:
    event_based_sync:
      # account:
      #  local_profile: "<ROLE-NAME>"
      kinesis_stream_arn: <OUTPUT-FROM-CLOUDFORMATION-STACK>
```

--------------------------------

### Add table_options to aws_support_trusted_advisor_checks

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This enhancement provides `table_options` support for the `aws_support_trusted_advisor_checks` table, improving its usability. This change was included in v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/support/trusted_advisor_checks.go
```

--------------------------------

### Update aws-sdk-go-v2 Monorepo (Go)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

This entry represents routine updates to the AWS SDK for Go v2 monorepo. These updates often include performance improvements, bug fixes, and compatibility enhancements for various AWS services.

```go
github.com/aws/aws-sdk-go-v2
```

--------------------------------

### AWS InspectorV2 ListCoverage Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for listing AWS InspectorV2 coverage. Prohibits NextToken and enforces a MaxResults range of 1-200.

```yaml
aws_inspector2_coverage:
  ListCoverage:
    - <InspectorV2.ListCoverage> # NextToken is prohibited. MaxResults should be in range [1-200].
```

--------------------------------

### AWS Environment Variables for Authentication (Linux/Mac)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Sets AWS credentials and session token using environment variables. This method is respected by the CloudQuery AWS plugin during authentication.

```bash
export AWS_ACCESS_KEY_ID={Your AWS Access Key ID}
export AWS_SECRET_ACCESS_KEY={Your AWS secret access key}
export AWS_SESSION_TOKEN={Your AWS session token}

```

--------------------------------

### Update CloudQuery Plugin SDK (Go)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

This signifies an update to the CloudQuery Plugin SDK for Go. The SDK provides the necessary tools and interfaces for developing CloudQuery plugins, and updates often bring new functionalities or bug fixes.

```go
github.com/cloudquery/plugin-sdk/v4
```

--------------------------------

### AWS RDS Engine Version Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for describing AWS RDS engine versions. Prohibits Marker and enforces a MaxRecords range of 20-100.

```yaml
aws_rds_engine_versions:
  DescribeDBEngineVersions:
    - <RDS.DescribeDBEngineVersions> # Marker is prohibited. MaxRecords should be in range [20-100].
```

--------------------------------

### AWS Plugin: Source Credentials from Named Profile for AWS Organizations Admin Account

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This snippet shows how to source credentials from a named profile in the shared configuration file for the admin account when configuring AWS Organizations.

```yaml
    org:
      member_role_name: cloudquery-ro
      admin_account:
        local_profile: <Named-Profile>

```

--------------------------------

### AWS SSM Inventory Entries Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for listing AWS SSM inventory entries. Prohibits NextToken, requires InstanceId and TypeName, and enforces a MaxResults range of 1-50.

```yaml
aws_ssm_inventory_entries:
  ListInventoryEntries:
    # NextToken is prohibited. MaxResults should be in range [1-50].
    # InstanceId and TypeName are required.
    - <SSM.ListInventoryEntries>
```

--------------------------------

### AWS SSM Describe Sessions Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for describing AWS SSM sessions. Prohibits NextToken and enforces a MaxResults range of 1-200.

```yaml
aws_ssm_sessions:
  DescribeSessions:
    - <SSM.DescribeSessions> # NextToken is prohibited. MaxResults should be in range [1-200].
```

--------------------------------

### Add AWS Inspector2 CIS Scans Resources

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

Introduces new resources for AWS Inspector2 CIS scans, including scan results, scans, and target resource aggregations. This enhancement improves security and compliance monitoring capabilities.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/inspector2
```

--------------------------------

### S3 Notification Configuration for SQS

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Configures S3 bucket notifications to send messages to an SQS queue for specific events, like object creation.

```json
{
    "QueueConfigurations": [
        {
            "QueueArn": "arn:aws:sqs:<REGION>:<ACCOUNT_ID>:<REPLACE_WITH_QUEUE_NAME>",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}
```

--------------------------------

### AWS Credentials File with Named Profile

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Defines a named profile within the AWS credentials file, allowing the CloudQuery plugin to use specific credentials by referencing the profile name.

```ini
[myprofile]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

```

--------------------------------

### AWS Security Hub Findings Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for retrieving AWS Security Hub findings. Prohibits NextToken and enforces a MaxResults range of 1-100.

```yaml
aws_securityhub_findings:
  GetFindings:
    - <SecurityHub.GetFindings> # NextToken is prohibited. MaxResults should be in range [1-100].
```

--------------------------------

### Improve region selection for all regions sync

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This improvement enhances the region selection logic when syncing data from all AWS regions, leading to more efficient and accurate data extraction. This change was part of release v27.19.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/service/region.go
```

--------------------------------

### SQL Query: Find all public-facing load balancers in AWS

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

This SQL query retrieves all AWS Elastic Load Balancing v2 (ELbv2) load balancers that are configured as 'internet-facing'.

```sql
SELECT * FROM aws_elbv2_load_balancers WHERE scheme = 'internet-facing';

```

--------------------------------

### Set AWS Profile via Environment Variable (Linux/Mac)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Sets the AWS_PROFILE environment variable to specify which named profile from the credentials file the CloudQuery plugin should use for authentication.

```bash
export AWS_PROFILE=myprofile

```

--------------------------------

### CloudQuery Configuration with Local Profile

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Configures the CloudQuery plugin to use a specific named profile ('myprofile') from the AWS credentials file via the `local_profile` setting within the CloudQuery configuration.

```yaml
accounts:
  id: <account_alias>
  local_profile: myprofile

```

--------------------------------

### Add vault_state column to aws_backup_vaults

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=9

This enhancement adds the `vault_state` column of type `utf8` to the `aws_backup_vaults` table, providing information about the backup vault's status. This was included in release v27.18.0.

```go
github.com/cloudquery/cloudquery/plugin/source/aws/client/backup/vaults.go
```

--------------------------------

### AWS Route53 Hosted Zone Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for listing AWS Route53 hosted zones. Prohibits NextToken, DelegationSetId, and HostedZoneType, with a MaxResults range of 1-100.

```yaml
aws_route53_hosted_zones:
  ListHostedZones: 
    # NextToken, DelegationSetId and HostedZoneType are prohibited. MaxResults should be in range [1-100].
    - <Route53.GetHostedZone>
```

--------------------------------

### Update CloudQuery Codegen Module (Go)

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/versions_page=5

This indicates an update to the CloudQuery codegen module, which is essential for generating CloudQuery plugins and resources. Updates to this module can include new features or improvements to the code generation process.

```go
github.com/cloudquery/codegen
```

--------------------------------

### Test AWS CLI Authentication

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Verifies if the AWS Command Line Interface is authenticated and can access AWS services. This is a prerequisite for non-interactive authentication.

```bash
aws ec2 describe-regions

```

--------------------------------

### AWS Service Quotas Options

Source: https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs

Specifies options for listing AWS service quotas. Prohibits NextToken and enforces a MaxResults range of 1-100 for ListServices.

```yaml
aws_servicequotas_services:
  ListServices: <ServiceQuota.ListServices> # NextToken is prohibited. MaxResults should be in range [1-100].
  ListServiceQuotas:
    - <ServiceQuota.ListServiceQuotas>
```
Tokens:
Raw