### RisingWave Model Configuration Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Example of using the `config()` macro in a dbt model to set RisingWave-specific options. ```sql {{ config( materialized='materialized_view', schema_authorization='role_name', streaming_parallelism=4, streaming_parallelism_for_backfill=2, streaming_max_parallelism=8, enable_serverless_backfill=true, background_ddl=true, sql_header='SET query_mode = local;', indexes=[ {'columns': ['col1'], 'include': ['col2'], 'distributed_by': ['col1']} ] ) }} ``` -------------------------------- ### Install dbt-risingwave Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Install the dbt-risingwave adapter using pip. ```shell python3 -m pip install dbt-risingwave ``` -------------------------------- ### RisingWave dbt Profile Configuration Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Example of a dbt profiles.yml file demonstrating connection and session settings for RisingWave. Includes both development and production configurations. ```yaml default: outputs: dev: type: risingwave host: localhost port: 4566 user: root pass: "" dbname: dev schema: public # Optional streaming session settings streaming_parallelism: 4 streaming_parallelism_for_backfill: 2 streaming_max_parallelism: 8 enable_serverless_backfill: true # Optional connection tuning keepalives_idle: 30 connect_timeout: 10 retries: 3 # Optional SSL/TLS sslmode: require sslcert: "~/.certs/client-cert.pem" sslkey: "~/.certs/client-key.pem" sslrootcert: "~/.certs/ca-cert.pem" prod: type: risingwave host: prod-risingwave.example.com port: 4566 user: analyst pass: "${RISINGWAVE_PASSWORD}" dbname: analytics schema: transformations streaming_parallelism: 16 streaming_max_parallelism: 32 target: dev ``` -------------------------------- ### RisingWave Profile Configuration Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md Example of a dbt profiles.yml file for connecting to RisingWave. It includes basic connection details and optional streaming session settings. ```yaml default: outputs: dev: type: risingwave host: localhost port: 4566 user: root dbname: dev schema: public streaming_parallelism: 4 streaming_parallelism_for_backfill: 2 enable_serverless_backfill: true ``` -------------------------------- ### Example Materialized View Model Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md This is an example of a dbt model SQL file that configures a model to be materialized as a materialized_view in RisingWave. ```sql -- Model SQL {{ config(materialized='materialized_view') }} SELECT * FROM {{ ref('events') }} ``` -------------------------------- ### Source Definition Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Defines a source for Kafka events. ```sql {{ config( materialized='source' ) }} CREATE SOURCE kafka_events WITH ( connector = 'kafka', brokers = 'kafka:9092', topic = 'events', key_encoding = 'offset' ) ROW FORMAT JSON; ``` -------------------------------- ### Basic Materialized View Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Example of a basic materialized view configuration. ```sql {{ config( materialized='materialized_view' ) }} SELECT * FROM {{ ref('events') }} ``` -------------------------------- ### Function Configuration Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Configure a dbt function with its language, UDF server details, retry behavior, async/batch options, and volatility. ```yaml functions: - name: calculate_price description: Calculate price based on input config: language: javascript # javascript, python, or sql (default) # Python-specific link: http://udf-server:8815 remote_name: price_calc always_retry_on_network_error: false # JavaScript-specific async options async: true batch: false always_retry_on_network_error: true # Volatility (mapped to IMMUTABLE, STABLE, VOLATILE) volatility: deterministic # deterministic, stable, non-deterministic arguments: - name: base_price data_type: float8 - name: multiplier data_type: integer returns: data_type: float8 ``` -------------------------------- ### Source Definition in dbt Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Example dbt model for creating a Kafka source in RisingWave. ```sql {{ config(materialized='source') }} CREATE SOURCE kafka_events WITH ( connector = 'kafka', brokers = 'kafka:9092', topic = 'events' ) ROW FORMAT JSON; ``` -------------------------------- ### Example Usage of Render SQL Header Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Demonstrates how to apply the `risingwave__render_sql_header` macro in a model's configuration to set custom SQL and session parameters. ```sql -- Model SQL {{ config( materialized='materialized_view', sql_header='SET query_mode = local;', streaming_parallelism=4, background_ddl=true ) }} SELECT * FROM {{ ref('events') }} ``` -------------------------------- ### RisingWaveAdapter Sleep Method Usage Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/03-adapter-and-relation.md Demonstrates how to use the adapter.sleep method within a dbt macro to pause execution. ```sql -- In a dbt macro {% set sleep_result = adapter.sleep(10) %} -- Execution pauses for 10 seconds, then continues ``` -------------------------------- ### Function Signature with Type Annotations Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md Example of a function signature from connections.py demonstrating the use of Optional and Dict type hints for parameters and return types. ```python def _super_open(cls, connection, extra_kwargs: Optional[Dict[str, str]] = None): -> Connection ``` -------------------------------- ### Complete RisingWave Profile Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md A comprehensive example of the dbt profiles.yml file for RisingWave, including connection details, optional streaming settings, and connection tuning parameters. ```yaml default: outputs: dev: type: risingwave host: 127.0.0.1 port: 4566 user: root pass: "" dbname: dev schema: public # Optional streaming session settings streaming_parallelism: 4 streaming_parallelism_for_backfill: 2 streaming_max_parallelism: 8 enable_serverless_backfill: true # Optional connection tuning keepalives_idle: 30 connect_timeout: 10 retries: 3 target: dev ``` -------------------------------- ### dbt Materialized View Configuration with Indexes Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/03-adapter-and-relation.md Example of how to configure a materialized view in dbt, specifying columns, include columns, and distributed by columns for an index. ```sql {{ config( materialized='materialized_view', indexes=[ { 'columns': ['user_id'], 'include': ['name', 'email'], 'distributed_by': ['user_id'] } ] ) }} SELECT * FROM {{ ref('users') }} ``` -------------------------------- ### Example Macro Error Reporting Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md An example of a runtime error that might occur within a dbt macro, specifically when attempting to create an index with no columns. ```text Runtime Error in macro risingwave__get_create_index_sql (macros/adapters.sql) Invalid operation, attempting to create an index with no columns. ``` -------------------------------- ### RisingWaveConnectionManager open Method Signature Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Defines the signature for the open method, which establishes and configures a database connection. It handles connection setup and session configuration. ```python @classmethod def open(cls, connection: Connection) -> Connection ``` -------------------------------- ### Create Connection Statement Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md This snippet demonstrates how to create a connection, for example, a Kafka connection, using the `materialized='connection'` configuration. It specifies the connection type and its parameters. ```sql {{ config(materialized='connection') }} CREATE CONNECTION kafka_conn WITH ( type = 'kafka', brokers = 'kafka:9092' ); ``` -------------------------------- ### Sink Definition Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Defines a sink for sending data to an Iceberg table. ```sql {{ config( materialized='sink' ) }} CREATE SINK iceberg_sink FROM {{ ref('events') }} WITH ( connector = 'iceberg', database.tables = 'events', warehouse.path = 's3://my-bucket/warehouse' ); ``` -------------------------------- ### RisingWave Index Configuration Validation Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md Demonstrates how dbt_common.exceptions.DbtRuntimeError is raised when validating an index configuration with no columns. This ensures that indexes are created with valid parameters. ```python from dbt.adapters.risingwave.relation_configs import RisingWaveIndexConfig from dbt_common.exceptions import DbtRuntimeError # This will raise DbtRuntimeError try: RisingWaveIndexConfig.from_dict({ 'columns': [], # Error: no columns 'include': [], 'distributed_by': [] }) except DbtRuntimeError as e: print(e) # "Indexes require at least one column..." ``` -------------------------------- ### Generated RisingWave Index SQL Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Example of the SQL generated by the dbt-risingwave adapter for creating an index with INCLUDE and DISTRIBUTED BY clauses. ```sql CREATE INDEX IF NOT EXISTS "__dbt_index_mv_user_id" ON mv (user_id) INCLUDE (name, email) DISTRIBUTED BY (user_id); ``` -------------------------------- ### Generated SQL for DROP INDEX Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Example of the SQL statement generated by the risingwave__get_drop_index_sql macro for dropping an index. ```sql DROP INDEX IF EXISTS "database"."schema"."__dbt_index_mv_user_id"; ``` -------------------------------- ### Parsing Model Config for RisingWave Index Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md Example of how a dbt model configuration for an index is parsed into a `RisingWaveIndexConfig` object. This demonstrates the mapping from YAML to the Python dataclass. ```yaml # In model config() indexes: - columns: ['user_id'] include: ['name', 'email'] distributed_by: ['user_id'] ``` ```python RisingWaveIndexConfig( name="", column_names=('user_id',), include_columns=('name', 'email'), distributed_by_columns=('user_id',) ) ``` -------------------------------- ### Python Package Entry Point for dbt Adapters Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Defines the entry point in setup.py for dbt to discover and load the RisingWave adapter plugin. ```yaml entry_points: dbt.adapters: risingwave = dbt.adapters.risingwave:Plugin ``` -------------------------------- ### dbt Core Entry Point Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/01-overview.md Shows how to import the Plugin object from the dbt-risingwave adapter for use within dbt Core. ```python from dbt.adapters.risingwave import Plugin ``` -------------------------------- ### Core Module Structure Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/01-overview.md Illustrates the directory structure of the dbt-risingwave adapter, highlighting key Python files and their roles within the project. ```text dbt/adapters/risingwave/ ├── __init__.py # Plugin entry point, exports Plugin, RisingWaveAdapter, RisingWaveCredentials ├── __version__.py # Version number ├── impl.py # RisingWaveAdapter class (main adapter implementation) ├── connections.py # Connection management and credential configuration ├── relation.py # Relation type handling and configuration └── relation_configs/ ├── __init__.py # Exports config classes ├── index.py # Index configuration for materialized views └── materialized_view.py # Materialized view configuration ``` -------------------------------- ### SQL Scalar Function Definition Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Example of defining a SQL scalar function in dbt for RisingWave. ```sql -- functions/double.sql select price * 2 -- functions/double.yml functions: - name: double arguments: - name: price data_type: float8 returns: data_type: float8 ``` -------------------------------- ### Sink Definition in dbt Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Example dbt model for creating an Iceberg sink from a RisingWave materialized view. ```sql {{ config(materialized='sink') }} CREATE SINK iceberg_sink FROM {{ ref('events') }} WITH ( connector = 'iceberg', warehouse.path = 's3://bucket/warehouse' ); ``` -------------------------------- ### dbt-risingwave Architecture Overview Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/00-index.md Illustrates the layered architecture of the dbt-risingwave adapter, including credentials, connection, adapter, relation types, configuration, and macro layers. ```text dbt-risingwave ├── Credentials Layer (RisingWaveCredentials) │ └── Connection Layer (RisingWaveConnectionManager) │ └── Session Configuration (streaming settings) │ └── SQL Execution (psycopg2) │ ├── Adapter Layer (RisingWaveAdapter) │ ├── Extends PostgresAdapter (SQL generation) │ ├── Overrides ConnectionManager │ └── Overrides Relation type (RisingWaveRelation) │ ├── Relation Types (RisingWaveRelationType) │ ├── Standard SQL (table, view, cte, materialized_view) │ ├── RisingWave-native (source, sink, connection, subscription) │ └── Special (function, external) │ ├── Configuration Layer │ ├── RisingWaveMaterializedViewConfig (with indexes) │ ├── RisingWaveIndexConfig (with validation) │ └── Change detection (RisingWaveIndexConfigChange) │ └── Macro Layer (Jinja2) ├── Adapter macros (risingwave__*) ├── Materialization macros └── Override PostgreSQL defaults ``` -------------------------------- ### JavaScript Scalar Function Definition Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Example of defining an asynchronous JavaScript scalar function in dbt for RisingWave. ```sql -- functions/async_fetch.sql export async function async_fetch(url) { const response = await fetch(url); return response.json(); } -- functions/async_fetch.yml functions: - name: async_fetch config: language: javascript async: true arguments: - name: url data_type: varchar returns: data_type: jsonb ``` -------------------------------- ### Get Maximum Relation Name Length Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Returns the maximum allowed length for relation names in RisingWave. ```python def relation_max_name_length(self): return 1024 ``` -------------------------------- ### RisingWave Adapter Inheritance Chain Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Illustrates the inheritance hierarchy of the RisingWaveAdapter, showing its relationship with dbt's base Adapter and the PostgresAdapter. ```text dbt.adapters.base.Adapter └── PostgresAdapter (from dbt-postgres) └── RisingWaveAdapter ``` -------------------------------- ### Global Serverless Backfill Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Enable serverless backfills globally in the `dbt_project.yml` file. ```yaml models: my_project: +enable_serverless_backfill: true ``` -------------------------------- ### RisingWaveRelation Get Relation Type Property Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/03-adapter-and-relation.md Provides access to the RisingWaveRelationType enum, which defines the relation types supported by this class. ```python @classproperty def get_relation_type(cls) -> Type[RisingWaveRelationType]: return RisingWaveRelationType ``` -------------------------------- ### Render SQL Header Macro Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Use this macro to prepend custom SQL and RisingWave session settings to model DDL statements. It supports configurations like `sql_header`, `background_ddl`, `streaming_parallelism`, and `enable_serverless_backfill`. ```jinja2 {% macro risingwave__render_sql_header() -%} {%- set header_parts = [] -%} {%- set user_header = config.get("sql_header", none) -%} {%- if user_header is not none -%} {%- do header_parts.append(user_header) -%} {%- endif -%} {%- set background_ddl = config.get("background_ddl", false) -%} {%- if background_ddl -%} {%- do header_parts.append("set background_ddl = true;") -%} {%- endif -%} {%- set streaming_parallelism = config.get("streaming_parallelism", none) -%} {%- if streaming_parallelism is not none -%} {%- do header_parts.append("set streaming_parallelism = " ~ streaming_parallelism ~ ";") -%} {%- endif -%} {%- set streaming_parallelism_for_backfill = config.get("streaming_parallelism_for_backfill", none) -%} {%- if streaming_parallelism_for_backfill is not none -%} {%- do header_parts.append("set streaming_parallelism_for_backfill = " ~ streaming_parallelism_for_backfill ~ ";") -%} {%- endif -%} {%- set streaming_max_parallelism = config.get("streaming_max_parallelism", none) -%} {%- if streaming_max_parallelism is not none -%} {%- do header_parts.append("set streaming_max_parallelism = " ~ streaming_max_parallelism ~ ";") -%} {%- endif -%} {%- set enable_serverless_backfill = config.get("enable_serverless_backfill", none) -%} {%- if enable_serverless_backfill is not none -%} {%- do header_parts.append("set enable_serverless_backfill = " ~ enable_serverless_backfill | lower ~ ";") -%} {%- endif -%} {{- header_parts | join("\n") -}} {%- endmacro %} ``` -------------------------------- ### Generated SQL for Alter Relation Comment Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Example of the SQL generated by the risingwave__alter_relation_comment macro, showing how comments with escaped single quotes are handled. ```sql COMMENT ON TABLE schema.table_name IS 'This is a comment with \'\'quotes\'\' escaped'; ``` -------------------------------- ### dbt-risingwave Main Module Exports Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md Lists the primary classes and entry points exported by the dbt-risingwave adapter for integration with dbt Core. Includes adapter, connection manager, credentials, and plugin. ```python from dbt.adapters.risingwave import ( RisingWaveAdapter, RisingWaveConnectionManager, RisingWaveCredentials, Plugin ) from dbt.adapters.risingwave.relation import ( RisingWaveRelation, RisingWaveRelationType ) from dbt.adapters.risingwave.relation_configs import ( RisingWaveIndexConfig, RisingWaveIndexConfigChange, RisingWaveMaterializedViewConfig, RisingWaveMaterializedViewConfigChangeCollection ) ``` -------------------------------- ### RisingWaveAdapter Methods Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Provides methods for interacting with the RisingWave database, including executing SQL, managing relations, and pausing execution. ```APIDOC ## RisingWaveAdapter **Location:** `dbt/adapters/risingwave/impl.py` Main adapter class. Inherits from `PostgresAdapter`. ### Methods **`sleep(seconds: float) -> None`** — Pause execution for given seconds (available in Jinja2) ```python @available @classmethod def sleep(cls, seconds): time.sleep(seconds) ``` **Inherited from PostgresAdapter:** - `execute(sql)` — Execute SQL on current connection - `get_columns_in_relation(relation)` — Fetch column metadata - `list_relations_without_caching(schema_relation)` — List relations in a schema - `get_relation_without_caching(relation)` — Get single relation metadata - `drop_relation(relation)` — Drop a relation - `create_schema(relation)` — Create a schema - `execute_macros()` — Execute dbt macros ``` -------------------------------- ### RisingWave Materialized View Configuration Class Methods Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Provides class methods for creating and parsing RisingWave materialized view configurations from various sources like dbt RelationConfig, database metadata, or dictionaries. ```python @classmethod def from_config(cls, relation_config: RelationConfig) -> Self: """Create from dbt RelationConfig""" @classmethod def from_relation_results(cls, relation_results: RelationResults) -> Self: """Create from database metadata""" @classmethod def from_dict(cls, config_dict: dict) -> Self: """Create from dictionary""" @classmethod def parse_config(cls, relation_config: RelationConfig) -> Dict: """Extract config to dictionary""" @classmethod def parse_relation_results(cls, relation_results: RelationResults) -> dict: """Extract database state to dictionary""" ``` -------------------------------- ### Get Columns in Relation Macro Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Retrieves column metadata for a relation by querying the `information_schema.columns` table. Returns an Agate table with specific column details. ```jinja2 {% macro risingwave__get_columns_in_relation(relation) -%} {% call statement('get_columns_in_relation', fetch_result=True) %} select column_name, data_type, null as character_maximum_length, null as numeric_precision, null as numeric_scale from {{ relation.information_schema('columns') }} where table_name = '{{ relation.identifier }}' {% if relation.schema %} and table_schema = '{{ relation.schema }}' {% endif %} order by ordinal_position {% endcall %} {% set table = load_result('get_columns_in_relation').table %} {{ return(sql_convert_columns_in_relation(table)) }} {% endmacro %} ``` -------------------------------- ### Adapter-Specific Macro Naming Convention Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Illustrates the naming convention for adapter-specific macros, prefixed with the adapter name to avoid conflicts and enable inheritance. ```sql {% macro risingwave__render_sql_header() %} {% macro risingwave__list_relations_without_caching() %} {% macro risingwave__create_schema() %} ``` -------------------------------- ### Package Metadata Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Configures the essential metadata for the dbt-RisingWave package, including name, version, description, and author information. ```python setup( name="dbt-risingwave", version=_plugin_version(), # From __version__.py description="The RisingWave adapter plugin for dbt", long_description=README.read_text(), long_description_content_type="text/markdown", license="http://www.apache.org/licenses/LICENSE-2.0", keywords="dbt RisingWave", author="Dylan Chen", author_email="zilin@risingwave-labs.com", url="https://github.com/risingwavelabs/dbt-risingwave", ) ``` -------------------------------- ### Data Flow for Creating a Materialized View in dbt-risingwave Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/00-index.md Details the step-by-step process when a user runs 'dbt run' to create a materialized view, from initial dbt project parsing to SQL execution and post-hook processing. ```text User runs: dbt run 1. dbt reads dbt_project.yml and profiles.yml 2. Credential parsing: profiles.yml → RisingWaveCredentials 3. Plugin discovery: entry_points → dbt.adapters.risingwave:Plugin 4. Plugin loads: RisingWaveAdapter, RisingWaveConnectionManager, RisingWaveCredentials 5. Macro registration: macros/ → risingwave__* macros 6. For each model: a. Parse config() block b. Determine materialization: materialized_view c. Load materialized_view.sql template d. Render Jinja2: - Calls render_sql_header() → risingwave__render_sql_header() → Generates: SET background_ddl = true; SET streaming_parallelism = 4; ... - Compiles SQL query - Calls list_relations_without_caching() → checks if MV exists e. Execute SQL: - Opens connection → RisingWaveConnectionManager.open() → _super_open() → psycopg2.connect() + retries → _configure_session() → SET RW_IMPLICIT_FLUSH TO true; SET streaming_parallelism = ... - Runs generated CREATE MATERIALIZED VIEW SQL - If indexes configured, runs CREATE INDEX for each f. Run post-hooks (contracts, grants, etc.) g. Mark as success/failure 7. Close all connections ``` -------------------------------- ### RisingWaveRelationType Usage Example Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md Demonstrates how to check a relation's type against the RisingWaveRelationType enum and how to retrieve the string value of a type. Ensure the `RisingWaveRelationType` is imported before use. ```python from dbt.adapters.risingwave.relation import RisingWaveRelationType # Check a relation's type if relation.type == RisingWaveRelationType.MaterializedView: # Handle materialized view pass # Get the string value type_str = RisingWaveRelationType.Table.value # "table" ``` -------------------------------- ### Configure View for Zero Downtime Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/zero-downtime-rebuilds.md Use this configuration to enable zero-downtime rebuilds for views. This applies to the 'view' materialization. ```sql {{ config( materialized='view', zero_downtime={'enabled': true} ) }} select * from {{ ref('source_table') }} ``` -------------------------------- ### Serverless Backfill Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Enables serverless backfills for streaming queries. ```sql {{ config( materialized='materialized_view', enable_serverless_backfill=true ) }} SELECT * FROM {{ ref('historical_events') }} ``` -------------------------------- ### Get Relation Without Caching Macro Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Retrieves a relation's metadata from RisingWave catalogs without using the cache. Use this when you need the most up-to-date relation information. ```jinja2 {% macro risingwave__get_relation_without_caching(relation) %} {% set catalog_relations = risingwave__list_relations_without_caching(relation) %} {% for catalog_relation in catalog_relations %} {% if catalog_relation[1] == relation.identifier %} {{ return(api.Relation.create( database=catalog_relation[0], identifier=catalog_relation[1], schema=catalog_relation[2], type=catalog_relation[3] )) }} {% endif %} {% endfor %} {{ return(none) }} {% endmacro %} ``` -------------------------------- ### RisingWave Index Configuration Class Methods and Properties Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Provides methods for creating RisingWave index configurations from dictionaries, dbt model configs, or database metadata, and a property to convert to a dbt model config dictionary. ```python @classmethod def from_dict(cls, config_dict) -> RisingWaveIndexConfig: """Create from dictionary (columns are lowercased)""" @classmethod def parse_model_node(cls, model_node_entry: dict) -> dict: """Extract from dbt model config""" @classmethod def parse_relation_results(cls, relation_results_entry: agate.Row) -> dict: """Extract from database metadata""" @property def as_node_config(self) -> dict: """Convert to dbt model config dictionary""" ``` -------------------------------- ### Configure Raw SQL Sink DDL Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Use this configuration when you need full control over the sink statement. Omit the `connector` config, and the adapter will execute the SQL in the model as-is. ```sql {{ config(materialized='sink') }} create sink my_sink from my_mv with ( connector = 'blackhole' ) ``` -------------------------------- ### RisingWave Index Configuration with INCLUDE and DISTRIBUTED BY Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Configure RisingWave indexes in model configs, supporting INCLUDE and DISTRIBUTED BY clauses beyond standard Postgres adapter features. ```sql {{ config( materialized='materialized_view', indexes=[ {'columns': ['user_id'], 'include': ['name', 'email'], 'distributed_by': ['user_id']} ] ) }} ``` -------------------------------- ### Macro Helper Functions in Jinja2 Context Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Demonstrates common Jinja2 macro helper functions available within dbt, such as executing SQL statements, logging, type checking, and SQL generation utilities. ```jinja2 {# Statements #} {% call statement('name', fetch_result=True) %} SELECT * FROM table {% endcall %} {{ load_result('name').table }} {# Logging #} {% do log('message', info=true) %} {# Type checking #} {% if relation.type == 'materialized_view' %} {# SQL generation #} {{ relation }} {# Quoted relation name #} {{ adapter.quote(identifier) }} {# Quote an identifier #} {{ sql | strip_newlines }} {# Remove newlines from SQL #} ``` -------------------------------- ### Model Configuration for Background DDL Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Enable RisingWave background DDL for supported materializations like `materialized_view`, `table`, or `sink` on a per-model basis. The adapter waits for DDL completion. ```sql {{ config( materialized='materialized_view', background_ddl=true ) }} select * from {{ ref('events') }} ``` -------------------------------- ### Compare Materialized View Configurations for Changes Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/03-adapter-and-relation.md Compares the existing materialized view configuration with the desired configuration to identify necessary changes, particularly for indexes. This process informs the generation of ALTER statements for indexes without dropping and recreating the entire materialized view. ```python def get_materialized_view_config_change_collection( self, relation_results: RelationResults, relation_config: RelationConfig ) -> Optional[RisingWaveMaterializedViewConfigChangeCollection]: ``` ```sql -- Model config with indexes {{ config( materialized='materialized_view', indexes=[ {'columns': ['user_id'], 'include': ['name', 'email']} ] ) }} SELECT * FROM {{ ref('users') }} ``` -------------------------------- ### Version Definition Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Defines the version of the dbt-RisingWave adapter. This version is read by setup.py to avoid duplication. ```python version = "1.11.9" ``` -------------------------------- ### Table with Streaming Parallelism Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Configures a table with specific settings for streaming parallelism, including for backfills and maximum parallelism. ```sql {{ config( materialized='table', streaming_parallelism=8, streaming_parallelism_for_backfill=4, streaming_max_parallelism=16 ) }} SELECT * FROM {{ ref('large_dataset') }} ``` -------------------------------- ### Zero-Downtime Rebuild Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/04-configuration-reference.md Configures a model for zero-downtime rebuilds via table swapping. Requires passing `--vars 'zero_downtime: true'` at runtime. ```sql {{ config( materialized='materialized_view', zero_downtime={'enabled': true} ) }} SELECT * FROM {{ ref('events') }} ``` -------------------------------- ### RisingWave DBT Adapter Module Exports Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Lists the main components exported by the dbt.adapters.risingwave package, including connection managers, credentials, and the adapter itself. ```python __init__.py exports: ├── RisingWaveConnectionManager ├── RisingWaveCredentials ├── RisingWaveAdapter └── Plugin (AdapterPlugin instance) ``` -------------------------------- ### Enable Serverless Backfill Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Enable serverless backfills for streaming queries by setting enable_serverless_backfill to true in model configs or profiles. ```sql enable_serverless_backfill=true ``` -------------------------------- ### dbt Profiles Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Configure your dbt profiles.yml file to connect to RisingWave. Ensure RisingWave is running and accessible. ```yaml default: outputs: dev: type: risingwave host: 127.0.0.1 user: root pass: "" dbname: dev port: 4566 schema: public target: dev ``` -------------------------------- ### Create RisingWave Subscription Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Use `materialized='subscription'` to create a RisingWave subscription. This is useful for keeping the upstream log store available for cross-database materialized views managed from another target database. The subscription model SQL must render to the table or materialized view being subscribed. ```sql {{ config( materialized='subscription', retention='1D' ) }} {{ ref('events') }} ``` -------------------------------- ### Model Configuration for Serverless Backfill Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Enable serverless backfills for streaming queries on a per-model basis. This emits a SET statement before the model DDL. ```sql {{ config( materialized='materialized_view', enable_serverless_backfill=true ) }} select * from {{ ref('events') }} ``` -------------------------------- ### Internal Connection Establishment with Retry Logic Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md This internal method handles establishing a connection using psycopg2 with retry logic for operational errors. It merges credentials and extra keyword arguments for the connection. ```python def _super_open(cls, connection: Connection, extra_kwargs: Optional[Dict[str, str]] = None) -> Connection: pass ``` -------------------------------- ### Create Subscription Statement Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Use this snippet to create a subscription, which maintains a log of changes. It requires the `materialized='subscription'` configuration and references a source materialized view. ```sql {{ config(materialized='subscription') }} CREATE SUBSCRIPTION sub_name FROM {{ ref('mv_name') }}; ``` -------------------------------- ### Enable Background DDL Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Set background_ddl=true to allow supported materializations to submit background DDL while maintaining dbt semantics. ```sql background_ddl=true ``` -------------------------------- ### Compare Materialized View Configurations Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Compares materialized view configurations to detect index additions/removals and generate change collections for ALTER statements. ```python def get_materialized_view_config_change_collection( self, relation_results: RelationResults, relation_config: RelationConfig ) -> Optional[RisingWaveMaterializedViewConfigChangeCollection]: ``` -------------------------------- ### Model Configuration for SQL Header Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Prepend custom SQL statements before the main DDL for a model using the `sql_header` config. ```sql {{ config( materialized='table', sql_header='set query_mode = local;' ) }} select * from ... ``` -------------------------------- ### dbt-risingwave Repository File Organization Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/00-index.md Details the directory structure of the dbt-risingwave repository, showing the location of adapter code, macros, documentation, tests, and configuration files. ```text dbt-risingwave/ ├── dbt/adapters/risingwave/ │ ├── __init__.py # Plugin entry point │ ├── __version__.py # Version string │ ├── impl.py # RisingWaveAdapter │ ├── connections.py # RisingWaveConnectionManager, RisingWaveCredentials │ ├── relation.py # RisingWaveRelation, RisingWaveRelationType │ └── relation_configs/ │ ├── __init__.py │ ├── index.py # RisingWaveIndexConfig, RisingWaveIndexConfigChange │ └── materialized_view.py # RisingWaveMaterializedViewConfig │ ├── dbt/include/risingwave/ │ ├── macros/ │ │ ├── adapters.sql # Catalog, schema, column, comment, index macros │ │ ├── catalog.sql # get_catalog_relations, get_catalog │ │ └── materializations/ │ │ ├── materialized_view.sql │ │ ├── table.sql │ │ ├── view.sql │ │ ├── incremental.sql │ │ ├── connection.sql │ │ ├── source.sql │ │ ├── sink.sql │ │ ├── subscription.sql │ │ ├── table_with_connector.sql │ │ ├── functions/ │ │ │ └── scalar.sql # UDF materialization │ │ ├── test.sql │ │ └── grants.sql │ └── dbt_project.yml # Macro project metadata │ ├── docs/ │ ├── configuration.md # Configuration guide │ ├── functions.md # UDF documentation │ └── zero-downtime-rebuilds.md # Zero-downtime rebuild guide │ ├── tests/ │ └── e2e/ # End-to-end tests │ ├── basic_models/ │ ├── contracts/ │ ├── functions/ │ └── ... │ ├── setup.py # Package configuration ├── README.md # User guide └── ZERO_DOWNTIME_MV_README.md # Zero-downtime rebuild guide ``` -------------------------------- ### Configure RisingWave Session Settings Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md Applies RisingWave-specific session settings to an open psycopg2 connection based on provided credentials. This includes implicit flush and parallelism settings. ```python def _configure_session(handle, credentials: RisingWaveCredentials): pass ``` -------------------------------- ### Create Source Statement Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/05-macros-and-materializations.md Use this snippet to create a source object, such as a Kafka source, by defining its connector and topic. It requires the `materialized='source'` configuration. ```sql {{ config(materialized='source') }} CREATE SOURCE kafka_source WITH ( connector = 'kafka', brokers = 'kafka:9092', topic = 'events' ) ROW FORMAT JSON; ``` -------------------------------- ### Global Background DDL Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Enable background DDL globally in the `dbt_project.yml` file for supported dbt resource types. ```yaml models: my_project: +background_ddl: true ``` -------------------------------- ### Configure Materialized View Materialization Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Use this configuration to create a materialized view in RisingWave. This is the primary streaming materialization. ```sql {{ config(materialized='materialized_view') }} select * from {{ ref('events') }} ``` -------------------------------- ### Run Tests Matching a Pattern Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Execute tests that match a specific keyword pattern, like 'materialized_view', across all tests in the 'tests/' directory. This is useful for focusing on tests related to a particular feature or component. ```bash pytest tests/ -k "materialized_view" # Run tests matching a pattern ``` -------------------------------- ### List Temporary Objects Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/zero-downtime-rebuilds.md Run this dbt operation to list all preserved temporary objects created during zero-downtime rebuilds. ```bash dbt run-operation list_temp_objects ``` -------------------------------- ### Configure Indexes for Materialized Views Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Configure indexes for `materialized_view`, `table`, and `table_with_connector` using the `indexes` argument. The `on_configuration_change='apply'` setting allows dbt to apply index configuration changes automatically. ```sql {{ config( materialized='materialized_view', indexes=[{'columns': ['user_id']}], on_configuration_change='apply' ) }} select * from {{ ref('events') }} ``` -------------------------------- ### JavaScript Async Options Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/functions.md Configures a JavaScript scalar function with async, batch, and network retry options for RisingWave. ```yaml functions: - name: http_get_todo_name_js config: language: javascript async: true batch: false always_retry_on_network_error: false ``` -------------------------------- ### Open RisingWave Connection Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md Opens a connection to RisingWave and configures the session with RisingWave-specific settings. This method is called internally by dbt. ```python # dbt internally calls this when opening a connection connection = RisingWaveConnectionManager.open(connection) ``` -------------------------------- ### Configure Adapter-Managed Sink DDL Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/configuration.md Define sink connector settings directly in the model configuration. The adapter will automatically generate the `CREATE SINK` statement. ```sql {{ config( materialized='sink', connector='kafka', connector_parameters={ 'topic': 'orders', 'properties.bootstrap.server': '127.0.0.1:9092' }, data_format='plain', data_encode='json', format_parameters={} ) }} select * from {{ ref('orders_mv') }} ``` -------------------------------- ### Configure Materialized View with Default Cleanup Behavior Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/zero-downtime-rebuilds.md This configuration preserves temporary objects after a swap to prevent breaking downstream dependencies. Zero downtime is enabled. ```sql {{ config( materialized='materialized_view', zero_downtime={'enabled': true, 'immediate_cleanup': false} ) }} ``` -------------------------------- ### Configure Materialized View for Zero Downtime Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/zero-downtime-rebuilds.md Use this configuration to enable zero-downtime rebuilds for materialized views. Ensure your RisingWave version is 2.2 or later. ```sql {{ config( materialized='materialized_view', zero_downtime={'enabled': true} ) }} select * from {{ ref('source_table') }} ``` -------------------------------- ### Plugin Class Registration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Registers the RisingWave adapter with dbt using the AdapterPlugin class. Specifies the adapter, credentials, include path, and dependencies. ```python from dbt.adapters.base import AdapterPlugin from dbt.include import risingwave Plugin = AdapterPlugin( adapter=RisingWaveAdapter, credentials=RisingWaveCredentials, include_path=risingwave.PACKAGE_PATH, dependencies=["postgres"], ) ``` -------------------------------- ### Run Specific Test Project Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Execute end-to-end tests for a particular project, such as 'basic_models'. This command targets a specific subdirectory within the 'tests/e2e/' directory. ```bash pytest tests/e2e/basic_models/ # Run specific test project ``` -------------------------------- ### RisingWave Python Function Creation Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/docs/functions.md Displays the SQL statement for creating external Python UDFs in RisingWave, specifying the remote name and link. ```sql CREATE FUNCTION IF NOT EXISTS ... AS 'remote_name' USING LINK 'http://host:port'; ``` -------------------------------- ### RisingWave Materialized View Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Represents the configuration for a RisingWave materialized view, including table name, query, and indexes. ```python @dataclass(frozen=True, eq=True, unsafe_hash=True) class RisingWaveMaterializedViewConfig( RelationConfigBase, RelationConfigValidationMixin ): table_name: str = "" query: str = "" indexes: List[RisingWaveIndexConfig] = field(default_factory=list) ``` -------------------------------- ### Common Python Type Hints Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/06-types-and-errors.md A list of commonly used Python type hints and their meanings, useful for understanding type annotations in the adapter. ```python Optional[T] # T or None Dict[K, V] # Dictionary with key type K, value type V List[T] # List of type T tuple[str, ...] # Tuple of strings (PEP 646) Callable[[A, B], R] # Function taking A, B and returning R Union[A, A] # Either type A or B ``` -------------------------------- ### Python Version Support Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/07-plugin-system.md Specifies the compatible Python versions for the dbt-RisingWave adapter, ensuring compatibility with dbt-core requirements. ```python python_requires=">=3.9", classifiers=[ "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", ] ``` -------------------------------- ### RisingWave Index Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/08-api-summary.md Represents the configuration for a RisingWave index, including column names and distribution columns. ```python @dataclass(frozen=True, eq=True, unsafe_hash=True) class RisingWaveIndexConfig(RelationConfigBase, RelationConfigValidationMixin): name: str = field(default="", hash=False, compare=False) column_names: tuple[str, ...] = field(default_factory=tuple, hash=True) include_columns: tuple[str, ...] = field(default_factory=tuple, hash=True) distributed_by_columns: tuple[str, ...] = field(default_factory=tuple, hash=True) ``` -------------------------------- ### Zero-Downtime Rebuilds Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/README.md Configure zero-downtime rebuilds for materialized_view and view materializations using zero_downtime={'enabled': true}. ```sql {{ config( materialized='materialized_view', zero_downtime={'enabled': true} ) }} ``` -------------------------------- ### SQL Statements for Session Configuration Source: https://github.com/risingwavelabs/dbt-risingwave/blob/main/_autodocs/02-credentials-and-connection.md SQL commands executed to configure the RisingWave session based on the provided profile settings. These include implicit flush and parallelism settings. ```sql SET RW_IMPLICIT_FLUSH TO true; SET streaming_parallelism = 4; SET streaming_parallelism_for_backfill = 2; SET enable_serverless_backfill = true; ```