### Example Credentials JSON for Integration Test Source: https://github.com/dataform-co/dataform/blob/main/contributing.md An example of the credentials JSON file format required for running CLI integration tests against a GCP project. Ensure 'projectId', 'credentials', and 'location' are correctly set. ```json { "projectId": "my-gcp-project", "credentials": "{\"type\":\"service_account\",...}", "location": "US" } ``` -------------------------------- ### Install Bazelisk via NPM Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Install the correct Bazel version using Bazelisk via NPM. This is the recommended method for setting up the Bazel build tool. ```bash npm i -g @bazel/bazelisk ``` -------------------------------- ### Install Dataform CLI Source: https://github.com/dataform-co/dataform/blob/main/readme.md Install the Dataform CLI tool globally using npm. This command is used to run Dataform locally. ```bash npm i -g @dataform/cli ``` -------------------------------- ### Basic Notebook File Structure Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/notebook.md An example of a minimal Jupyter Notebook file structure. ```json # definitions/name.ipynb { "cells": [] } ``` -------------------------------- ### Creating a View using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Example of how to create a view using the `table` function and the `query` method from the Javascript API. ```APIDOC ## Creating a View using Javascript API ```js // definitions/file.js table("name", { type: "view" }).query("SELECT column FROM someTable") ``` **Note:** Methods in the `View` class can be accessed by the returned value of the `table` function. ``` -------------------------------- ### hasOutput Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. Deprecated in favor of OperationConfig.hasOutput. ```APIDOC ## hasOutput ### Description Declares that this action creates a dataset which should be referenceable as a dependency target, for example by using the `ref` function. ### Method hasOutput ### Parameters #### Path Parameters - **hasOutput** (boolean) - Required - Indicates if the action has output. ### Returns *this* ``` -------------------------------- ### Set Post-Operations for a View Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets a post-operation to run after the query is executed. This is often used for revoking temporary permissions granted to access source tables. The example demonstrates revoking BigQuery data viewer roles. ```javascript publish("example") .preOps(ctx => `GRANT `roles/bigquery.dataViewer ` ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE `roles/bigquery.dataViewer ` ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) ``` -------------------------------- ### Run Dataform CLI Help Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Execute the Dataform CLI to display default help information. Replace 'dataform' with './scripts/run' to use the project's CLI. ```bash ./scripts/run help ``` -------------------------------- ### Create Table using SQLX File Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Use a SQLX file to define a table. Ensure the config block specifies type: "table". ```sql -- definitions/name.sqlx config { type: "table" } SELECT 1 ``` -------------------------------- ### Define Notebook using Action Config Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/notebook.md Create a notebook action by specifying its filename in an actions configuration file. ```yaml # definitions/actions.yaml actions: - notebook: filename: name.ipynb ``` -------------------------------- ### Session.notebook Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a Notebook action. Available only in the `/definitions` directory. ```APIDOC ## Session.notebook ### Description Creates a Notebook action. Available only in the `/definitions` directory. ### Method `notebook(config: NotebookConfig): Notebook` ### Parameters #### Path Parameters - **config** (NotebookConfig) - Required - The configuration for the notebook. ### Returns - **Notebook** - The created Notebook object. ### See - [Notebook](Notebook) for examples on how to use. ``` -------------------------------- ### Create View using SQLX file Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Define a view using a SQLX file. Ensure the config block specifies type: "view". ```sql config { type: "view" } SELECT column FROM someTable ``` -------------------------------- ### Create View using Action Configs Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Define a view using an actions.yaml file and a corresponding SQL file. The YAML specifies the view and its filename. ```yaml actions: - view: filename: name.sql ``` ```sql SELECT column FROM someTable ``` -------------------------------- ### Create Table using Action Configs Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Define a table using an actions.yaml file, referencing a separate SQL file for the query. ```yaml # definitions/actions.yaml actions: - table: filename: name.sql ``` ```sql -- definitions/name.sql SELECT 1 ``` -------------------------------- ### Build VS Code Extension Package Source: https://github.com/dataform-co/dataform/blob/main/vscode/contributing.md Run this command in the main repository to generate the VS Code extension package (.vsix file). This is the first step for both web and CLI publishing routes. ```bash bazel run vscode:packager /tmp/dataform-package.vsix ``` -------------------------------- ### Create a Dataform Notebook Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a Notebook action. Available only in the `/definitions` directory. ```javascript dataform.notebook({ name: "my_notebook", notebookPath: "my_notebook.ipynb" }) ``` -------------------------------- ### Notebook ipynb Method Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/notebook.md Sets or overrides the contents of the notebook to run. It is generally recommended to use separate .ipynb files for notebooks instead of this method. ```APIDOC ## ipynb ### Description Sets or overrides the contents of the notebook to run. Not recommended in general; using separate `.ipynb` files for notebooks is preferred. ### Parameters #### Path Parameters - **contents** (object) - Required - The content of the notebook. ### Returns * [Notebook](_core_actions_notebook_.notebook.md) ``` -------------------------------- ### Run Unit Tests for @dataform/core Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Execute the unit tests specifically for the @dataform/core package using Bazel. ```bash bazel test //core/... ``` -------------------------------- ### Publish VS Code Extension via CLI Source: https://github.com/dataform-co/dataform/blob/main/vscode/contributing.md This method involves pasting the generated package into a specific directory and then using `vsce publish`. Note that this method is currently not working as expected. ```bash vsce publish ``` -------------------------------- ### Publish VS Code Extension via Web Source: https://github.com/dataform-co/dataform/blob/main/vscode/contributing.md After generating the package, upload it to the Visual Studio Marketplace for web publishing. ```bash https://marketplace.visualstudio.com/manage/publishers/dataform ``` -------------------------------- ### Session.publish Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a table, view, or incremental table. Available only in the `/definitions` directory. ```APIDOC ## Session.publish ### Description Creates a table, view, or incremental table. Available only in the `/definitions` directory. ### Method `publish(name: string, queryOrConfig?: Contextable | TableConfig | ViewConfig | IncrementalTableConfig | ILegacyTableConfig | any): Table | IncrementalTable | View` ### Parameters #### Path Parameters - **name** (string) - Required - The name of the table, view, or incremental table. - **queryOrConfig?** (Contextable | TableConfig | ViewConfig | IncrementalTableConfig | ILegacyTableConfig | any) - Optional - The query or configuration for the table, view, or incremental table. ### Returns - **Table | IncrementalTable | View** - The created Table, IncrementalTable, or View object. ### See - [Operation](Operation) for examples on how to make tables. - [View](View) for examples on how to make views. - [IncrementalTable](IncrementalTable) for examples on how to make incremental tables. ``` -------------------------------- ### Session.test Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a Test action. Available only in the `/definitions` directory. ```APIDOC ## Session.test ### Description Creates a Test action. Available only in the `/definitions` directory. ### Method `test(name: string): Test` ### Parameters #### Path Parameters - **name** (string) - Required - The name of the test. ### Returns - **Test** - The created Test object. ### See - [Test](Test) for examples on how to use. ``` -------------------------------- ### Build the Dataform CLI Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Build the Dataform CLI using Bazel. This command compiles most of the necessary components for the CLI. ```bash bazel build cli ``` -------------------------------- ### Define Notebook using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/notebook.md Create a notebook action programmatically using the Javascript API, specifying the notebook name and filename. ```javascript // definitions/file.js notebook("name", { filename: "name.ipynb" }) ``` -------------------------------- ### CLI Integration Test Command Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Run the CLI integration test against your GCP project using Bazel. Ensure you have commented out the specified dependency and updated constants in `cli/index_test.ts`. ```bash bazel test //cli:index_test ``` -------------------------------- ### Set Notebook Contents Directly Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/notebook.md Use the `ipynb` method to set or override the contents of a notebook. It is generally recommended to use separate `.ipynb` files instead. ```javascript notebook("name").ipynb({ "cells": [] }) ``` -------------------------------- ### Create a Dataform Test Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a Test action. Available only in the `/definitions` directory. ```javascript dataform.test("unique_test_name") ``` -------------------------------- ### Declare a BigQuery Table using SQLX Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/declaration.md Use a SQLX file to declare a BigQuery table as a data source. Ensure no SQL is present in the file, only the config block. ```sqlx config { type: "declaration" } -- Note: no SQL should be present. ``` -------------------------------- ### Declare a BigQuery Table using Action Configs Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/declaration.md Declare a BigQuery table by specifying it within the 'actions' block of an YAML configuration file. ```yaml actions: - declare: name: name ``` -------------------------------- ### Access Project Configuration Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Access the Dataform project configuration, including variables, from the global `dataform` object. ```javascript dataform.projectConfig.vars.myVariableName === "myVariableValue" ``` -------------------------------- ### Create a Test using SQLX Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/test.md Define a Dataform test action within a SQLX file. This method is suitable for embedding tests directly alongside SQL definitions. ```sql -- definitions/name.sqlx config { type: "test" } input "foo" { SELECT 1 AS bar } SELECT 1 AS bar ``` -------------------------------- ### Define Operation in YAML and SQL Files Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Define operations using a YAML file to specify the action and a corresponding SQL file for the operation's query. ```yaml actions: - operation: filename: name.sql ``` ```sql -- definitions/name.sql DELETE FROM dataset.table WHERE country = 'GB' ``` -------------------------------- ### query Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets the query to generate the table from. ```APIDOC ## query ### Description Sets the query to generate the table from. ### Method This is a method signature, not an HTTP endpoint. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None provided. ### Response #### Success Response * this - Returns the current instance for chaining. #### Response Example None provided. ``` -------------------------------- ### query Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets the query to generate the table from. ```APIDOC ## query ### Description Sets the query to generate the table from. ### Method N/A ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body - **query** (Contextable) - Required - The query to generate the table. ### Request Example N/A ### Response #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### Create a Dataform Table Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Creates a table, view, or incremental table. Available only in the `/definitions` directory. ```javascript dataform.publish("my_table", "SELECT 1") ``` -------------------------------- ### Query Repositories Created During Extreme Weather Source: https://github.com/dataform-co/dataform/blob/main/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb Executes a BigQuery query to select repositories created during extreme weather events from a specified table. ```python %%bigquery results --project dataform-open-source SELECT * FROM `dataform-open-source.dataform_examples.repositories_created_during_extreme_weather` ``` -------------------------------- ### Session.declare Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Declares the dataset as a Dataform data source. Available only in the `/definitions` directory. ```APIDOC ## Session.declare ### Description Declares the dataset as a Dataform data source. Available only in the `/definitions` directory. ### Method `declare(config: DeclarationConfig | any): Declaration` ### Parameters #### Path Parameters - **config** (DeclarationConfig | any) - Required - The configuration for the declaration. ### Returns - **Declaration** - The created Declaration object. ### See - [Declaration](Declaration) for examples on how to use. ``` -------------------------------- ### database Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the database (Google Cloud project ID) in which to create the output of this action. Deprecated in favor of ViewConfig.project. ```APIDOC ## database ### Description Sets the database (Google Cloud project ID) in which to create the output of this action. ### Method ▸ database(`database`: string): *this* ### Parameters #### Path Parameters - `database` (string) - Description not available ### Returns *this* ``` -------------------------------- ### database Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets the database (Google Cloud project ID) in which to create the output of this action. Deprecated in favor of TableConfig.project. ```APIDOC ## database ### Description Sets the database (Google Cloud project ID) in which to create the output of this action. ### Method database ### Parameters #### Path Parameters - **database** (string) - Required - The name of the database (Google Cloud project ID). ### Returns *this* ``` -------------------------------- ### Run Linting Checks Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Execute the linting script to check for any code style or linting errors in the project. ```bash ./scripts/lint ``` -------------------------------- ### Table Javascript API Usage Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Demonstrates how to create a table using the Javascript API. The `table` function returns an instance of the Table class, which has methods like `query` to define the table's data. ```APIDOC ## Table Javascript API ### Description This section shows how to use the Javascript API to define a table. ### Method `table(name: string, config?: TableConfig)` ### Parameters * **name** (string) - Required - The name of the table. * **config** (TableConfig) - Optional - Configuration options for the table. ### Usage Example ```js table("name", { type: "table" }).query("SELECT 1 AS TEST") ``` ### Methods Available on Returned Object * **query(sql: string)**: Defines the SQL query for the table. * **assertions(assertions: Assertion[])**: Defines assertions for the table. * **bigquery(config: BigQueryTableConfig)**: Configures BigQuery specific settings for the table. * **columns(columns: Column[])**: Defines the columns for the table. * **database(database: string)**: Sets the database for the table. * **dependencies(dependencies: string[])**: Sets dependencies for the table. * **description(description: string)**: Sets the description for the table. * **disabled(disabled: boolean)**: Disables the table. * **hermetic(hermetic: boolean)**: Sets the table to be hermetic. * **postOps(postOps: string[])**: Sets post-operations for the table. * **preOps(preOps: string[])**: Sets pre-operations for the table. * **schema(schema: string)**: Sets the schema for the table. * **setDependOnDependencyAssertions(value: boolean)**: Sets whether to depend on dependency assertions. * **tags(tags: string[])**: Sets tags for the table. * **type(type: "table" | "view" | "incremental" | "materialized_view")**: Sets the type of the table. ``` -------------------------------- ### Session.operate Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Defines a SQL operation. Available only in the `/definitions` directory. ```APIDOC ## Session.operate ### Description Defines a SQL operation. Available only in the `/definitions` directory. ### Method `operate(name: string, queryOrConfig?: Contextable | OperationConfig): Operation` ### Parameters #### Path Parameters - **name** (string) - Required - The name of the operation. - **queryOrConfig?** (Contextable | OperationConfig) - Optional - The query or configuration for the operation. ### Returns - **Operation** - The created Operation object. ### See - [operation](Operation) for examples on how to use. ``` -------------------------------- ### Create View using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Create a view programmatically using the Javascript API. The `query` method is accessed via the returned table object. ```javascript table("name", { type: "view" }).query("SELECT column FROM someTable") ``` -------------------------------- ### bigquery Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets BigQuery options for the action. Deprecated in favor of options available directly on ViewConfig. ```APIDOC ## bigquery ### Description Sets BigQuery options for the action. ### Method ▸ bigquery(`bigquery`: IBigQueryOptions): *this* ### Parameters #### Path Parameters - `bigquery` (IBigQueryOptions) - Description not available ### Returns *this* ``` -------------------------------- ### preOps Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets a pre-operation to run before the query is executed. Useful for temporary permission grants. ```APIDOC ## preOps ### Description Sets a pre-operation to run before the query is run. This is often used for temporarily granting permission to access source tables. ### Parameters - `pres` ([Contextable](../modules/_core_contextables_.md#contextable)‹[ITableContext](../interfaces/_core_contextables_.itablecontext.md), string | string[]›) - Description of the pre-operation. ### Returns *this* ``` -------------------------------- ### Define Operation in SQLX File Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Use a SQLX file to define custom operations. Specify the type as 'operations' in the config block. ```sql config { type: "operations" } DELETE FROM dataset.table WHERE country = 'GB' ``` -------------------------------- ### Session.projectConfig Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Stores the Dataform project configuration of the current Dataform project. Can be accessed via the `dataform` global variable. ```APIDOC ## Session.projectConfig ### Description Stores the Dataform project configuration of the current Dataform project. Can be accessed via the `dataform` global variable. ### Example ```js dataform.projectConfig.vars.myVariableName === "myVariableValue" ``` ``` -------------------------------- ### database Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets the database (Google Cloud project ID) in which to create the output of this action. Deprecated in favor of IncrementalTableConfig.project. ```APIDOC ## database ### Description Sets the database (Google Cloud project ID) in which to create the output of this action. ### Method ```typescript database(database: string): this ``` ### Parameters * `database` (string) - Required - The database (project ID) to use. ### Returns * `this` ``` -------------------------------- ### Create Table using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Utilize the Javascript API to programmatically define a table. Methods like `query` are accessed via the returned object. ```javascript // definitions/file.js table("name", { type: "table" }).query("SELECT 1 AS TEST") ``` -------------------------------- ### bigquery Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets BigQuery options for the action. Deprecated in favor of options available directly on TableConfig. ```APIDOC ## bigquery ### Description Sets BigQuery options for the action. ### Method bigquery ### Parameters #### Path Parameters - **bigquery** (IBigQueryOptions) - Required - The BigQuery options to set. ### Returns *this* ``` -------------------------------- ### dependencies Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets dependencies of the view. Deprecated in favor of ViewConfig.dependencies. ```APIDOC ## dependencies ### Description Sets dependencies of the view. ### Method ▸ dependencies(`value`: [Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]): *this* ### Parameters #### Path Parameters - `value` ([Resolvable](../modules/_core_contextables_.md#resolvable) | [Resolvable](../modules/_core_contextables_.md#resolvable)[]) - Description not available ### Returns *this* ``` -------------------------------- ### dependencies Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets dependencies of the table. Deprecated in favor of TableConfig.dependencies. ```APIDOC ## dependencies ### Description Sets dependencies of the table. ### Method dependencies ### Parameters #### Path Parameters - **value** ([Resolvable] | [Resolvable][]) - Required - The dependency or an array of dependencies. ### Returns *this* ``` -------------------------------- ### Create Assertion using SQLX File Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md Define an assertion directly within a SQLX file. The config block specifies the type as 'assertion' and the query follows. ```sqlx -- definitions/name.sqlx config { type: "assertion" } SELECT * FROM table WHERE a IS NULL ``` -------------------------------- ### Define a SQL Operation Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Defines a SQL operation. Available only in the `/definitions` directory. ```javascript dataform.operate("unique_operation_name", "SELECT 1") ``` -------------------------------- ### schema Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the schema (BigQuery dataset) for the output table. Deprecated in favor of ViewConfig.dataset. ```APIDOC ## schema ### Description Sets the schema (BigQuery dataset) in which to create the output of this action. ### Parameters - `schema` (string) - The name of the schema. ### Returns *this* ### Deprecated Deprecated in favor of [ViewConfig.dataset](configs#dataform-ActionConfig-ViewConfig). ``` -------------------------------- ### Set Pre-Operations for a View Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Use `preOps` to define operations that run before the view's query is executed. This is useful for temporary permission grants. ```javascript publish("example") .preOps(ctx => `GRANT "roles/bigquery.dataViewer" ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE "roles/bigquery.dataViewer" ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) ``` -------------------------------- ### bigquery Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets BigQuery options for the action. Deprecated in favor of options available directly on IncrementalTableConfig. ```APIDOC ## bigquery ### Description Sets BigQuery options for the action. ### Method ```typescript bigquery(bigquery: IBigQueryOptions): this ``` ### Parameters * `bigquery` (IBigQueryOptions) - Required - The BigQuery options. ### Returns * `this` ``` -------------------------------- ### type Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the type of the action. Deprecated; action type should be set in the constructor. ```APIDOC ## type ### Description Sets the type of the action. ### Parameters - `type` (TableType) - The type of the table. ### Returns *this* ### Deprecated Deprecated in favor of action type can being set in the configs passed to action constructor functions, for example `publish("name", { type: "table" })`. ``` -------------------------------- ### View Class Methods Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md This section details the available methods for the View class, which can be used to configure view actions when using the Javascript API. ```APIDOC ## View Class Methods This class provides methods to configure view actions when using the Javascript API. ### Methods * [assertions](_core_actions_view_.view.md#assertions) * [bigquery](_core_actions_view_.view.md#bigquery) * [columns](_core_actions_view_.view.md#columns) * [database](_core_actions_view_.view.md#database) * [dependencies](_core_actions_view_.view.md#dependencies) * [description](_core_actions_view_.view.md#description) * [disabled](_core_actions_view_.view.md#disabled) * [hermetic](_core_actions_view_.view.md#hermetic) * [materialized](_core_actions_view_.view.md#materialized) * [postOps](_core_actions_view_.view.md#postops) * [preOps](_core_actions_view_.view.md#preops) * [query](_core_actions_view_.view.md#query) * [schema](_core_actions_view_.view.md#schema) * [setDependOnDependencyAssertions](_core_actions_view_.view.md#setdependondependencyassertions) * [tags](_core_actions_view_.view.md#tags) * [type](_core_actions_view_.view.md#type) ``` -------------------------------- ### query Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the SQL query used to generate the view. ```APIDOC ## query ### Description Sets the query to generate the table from. ### Parameters - `query` ([Contextable](../modules/_core_contextables_.md#contextable)‹[ITableContext](../interfaces/_core_contextables_.itablecontext.md), string›) - The SQL query string. ### Returns *this* ``` -------------------------------- ### schema Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets the schema (BigQuery dataset) in which to create the output of this action. Deprecated in favor of TableConfig.dataset. ```APIDOC ## schema ### Description Sets the schema (BigQuery dataset) in which to create the output of this action. Deprecated in favor of [TableConfig.dataset](configs#dataform-ActionConfig-TableConfig). ### Method This is a method signature, not an HTTP endpoint. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None provided. ### Response #### Success Response * this - Returns the current instance for chaining. #### Response Example None provided. ``` -------------------------------- ### schema Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets the schema (BigQuery dataset) in which to create the output of this action. Deprecated in favor of OperationConfig.dataset. ```APIDOC ## schema ### Description Sets the schema (BigQuery dataset) in which to create the output of this action. ### Method schema ### Parameters #### Path Parameters - **schema** (string) - Required - The schema to set. ### Returns *this* ``` -------------------------------- ### Set Pre-operation for Table Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Use preOps to define operations that run before the main query. This is useful for temporary permission grants on source tables. Requires a Contextable function returning a SQL string. ```javascript publish("example") .preOps(ctx => `GRANT roles/bigquery.dataViewer ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE roles/bigquery.dataViewer ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) ``` -------------------------------- ### database Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets the database (Google Cloud project ID) in which to create the corresponding view for this operation. Deprecated in favor of OperationConfig.project. ```APIDOC ## database ### Description Sets the database (Google Cloud project ID) in which to create the corresponding view for this operation. ### Method database ### Parameters #### Path Parameters - **database** (string) - Required - The database project ID. ### Returns *this* ``` -------------------------------- ### dependencies Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets dependencies of the table. Deprecated in favor of OperationConfig.dependencies. ```APIDOC ## dependencies ### Description Sets dependencies of the table. ### Method dependencies ### Parameters #### Path Parameters - **value** ([Resolvable] | [Resolvable][]) - Required - The dependencies to set. ### Returns *this* ``` -------------------------------- ### Calculate Mention Ratio Source: https://github.com/dataform-co/dataform/blob/main/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb Calculates the 'mention_ratio' by dividing 'repository_count' by 'extreme_weather_count' and replaces any resulting NaN values with 0. ```python results["mention_ratio"] = (results["repository_count"] / results["extreme_weather_count"]).replace(np.nan, 0) ``` -------------------------------- ### description Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the description of this view. Deprecated in favor of ViewConfig.description. ```APIDOC ## description ### Description Sets the description of this view. ### Method ▸ description(`description`: string): *this* ### Parameters #### Path Parameters - `description` (string) - Description not available ### Returns *this* ``` -------------------------------- ### Display First Few Rows of Results Source: https://github.com/dataform-co/dataform/blob/main/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb Displays the first few rows of the 'results' DataFrame to inspect the queried data. ```python results.head() ``` -------------------------------- ### Test Class Methods Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/test.md The Test class provides methods to define and configure unit tests for SQL queries. These methods allow setting input data, expected outputs, and the dataset for the test. ```APIDOC ## Methods ### dataset Sets the schema (BigQuery dataset) in which to create the output of this action. #### Parameters * **ref** ([Resolvable](../modules/_core_contextables_.md#resolvable)) #### Returns * this --- ### expect Sets the expected output of the query to being tested against. #### Parameters * **contextableQuery** ([Contextable](../modules/_core_contextables_.md#contextable)‹[IActionContext](../interfaces/_core_contextables_.iactioncontext.md), string›) #### Returns * this --- ### input Sets the input query to unit test against. #### Parameters * **refName** (string | string[]) * **contextableQuery** ([Contextable](../modules/_core_contextables_.md#contextable)‹[IActionContext](../interfaces/_core_contextables_.iactioncontext.md), string›) #### Returns * this ``` -------------------------------- ### dependencies Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets dependencies of the incremental table. Deprecated in favor of IncrementalTableConfig.dependencies. ```APIDOC ## dependencies ### Description Sets dependencies of the incremental table. ### Method ```typescript dependencies(value: Resolvable | Resolvable[]): this ``` ### Parameters * `value` (Resolvable | Resolvable[]) - Required - The dependency or an array of dependencies. ### Returns * `this` ``` -------------------------------- ### Assertion Methods Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md This section details the methods available on the Assertion class for configuring various aspects of an assertion, such as its query, schema, and dependencies. Note that many of these methods are deprecated in favor of AssertionConfig options. ```APIDOC ## database ### Description Sets the database (Google Cloud project ID) in which to create the corresponding view for this assertion. ### Method `database(database: string): this` ### Parameters - **database** (string) - The database project ID. ### Returns - *this* --- ## dependencies ### Description Sets dependencies of the assertion. ### Method `dependencies(value: Resolvable | Resolvable[]): this` ### Parameters - **value** (Resolvable | Resolvable[]) - The dependencies for the assertion. ### Returns - *this* --- ## description ### Description Sets the description of this assertion. ### Method `description(description: string): this` ### Parameters - **description** (string) - The description for the assertion. ### Returns - *this* --- ## disabled ### Description If called with `true`, this action is not executed. The action can still be depended upon. Useful for temporarily turning off broken actions. ### Method `disabled(disabled: boolean): this` ### Parameters - **disabled** (boolean) - Whether to disable the assertion. ### Returns - *this* --- ## hermetic ### Description If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. ### Method `hermetic(hermetic: boolean): void` ### Parameters - **hermetic** (boolean) - Whether the assertion is hermetic. ### Returns - *void* --- ## query ### Description Sets the query to be run by the assertion. ### Method `query(query: AContextable): this` ### Parameters - **query** (AContextable) - The SQL query for the assertion. ### Returns - *this* --- ## schema ### Description Sets the schema (BigQuery dataset) in which to create the corresponding view for this assertion. ### Method `schema(schema: string): this` ### Parameters - **schema** (string) - The schema for the assertion. ### Returns - *this* --- ## tags ### Description Sets a list of user-defined tags applied to this action. ### Method `tags(value: string | string[]): this` ### Parameters - **value** (string | string[]) - The tags to apply to the assertion. ### Returns - *this* ``` -------------------------------- ### Define Assertion in Action Config File Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md Use an action configuration file (YAML) to declare an assertion and specify its filename. ```yaml # definitions/actions.yaml actions: - assertion: filename: name.sql ``` -------------------------------- ### postOps Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets a post-operation to run after the query is run. This is often used for revoking temporary permissions granted to access source tables. ```APIDOC ## postOps ### Description Sets a post-operation to run after the query is run. This is often used for revoking temporary permissions granted to access source tables. ### Example ```js // definitions/file.js publish("example") .preOps(ctx => `GRANT \`roles/bigquery.dataViewer\` ON TABLE ${ctx.ref("other_table")} TO \"group:automation@example.com\"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE \`roles/bigquery.dataViewer\` ON TABLE ${ctx.ref("other_table")} TO \"group:automation@example.com\"`) ``` ### Method ▸ postOps(`posts`: [Contextable](../modules/_core_contextables_.md#contextable)‹[ITableContext](../interfaces/_core_contextables_.itablecontext.md), string | string[]›): *this* ### Parameters #### Path Parameters - `posts` ([Contextable](../modules/_core_contextables_.md#contextable)‹[ITableContext](../interfaces/_core_contextables_.itablecontext.md), string | string[]›) - Description not available ### Returns *this* ``` -------------------------------- ### schema Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets the schema (BigQuery dataset) in which to create the output of this action. Deprecated in favor of IncrementalTableConfig.dataset. ```APIDOC ## schema ### Description Sets the schema (BigQuery dataset) in which to create the output of this action. Deprecated in favor of IncrementalTableConfig.dataset. ### Method N/A ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body - **schema** (string) - Required - The schema (BigQuery dataset) for the output. ### Request Example N/A ### Response #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### Create a Test using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/test.md Define a Dataform test action using the Javascript API. This approach allows for programmatic test creation and integration within Javascript workflows. Methods like `input` and `expect` are accessed via the returned value of the `test` function. ```javascript // definitions/file.js test("name") .input("sample_data", `SELECT 1 AS bar`) .expect(`SELECT 1 AS bar`); publish("sample_data", { type: "table" }).query("SELECT 1 AS bar") ``` -------------------------------- ### Add New NPM Dependencies with Bazel Source: https://github.com/dataform-co/dataform/blob/main/contributing.md Add new NPM dependencies to the project using Bazel's Bazel run command for Yarn. This ensures dependencies are managed correctly within the Bazel build system. ```bash $bazel run @nodejs//:yarn add ... ``` -------------------------------- ### Set Schema (Deprecated) Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Deprecated in favor of TableConfig.dataset. Sets the BigQuery dataset for the output table. ```javascript schema("my_dataset") ``` -------------------------------- ### Define Incremental Table in SQLX Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Use this SQLX snippet to define an incremental table. Dataform will build it from scratch the first time and only insert/merge new rows on subsequent runs based on the configured conditions. ```sqlx config { type: "incremental" } -- This inserts `1` the first time running, and `2` on subsequent runs. SELECT ${when(incremental(), 1, 2) } ``` -------------------------------- ### Set Schema for a View (Deprecated) Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Use `schema` to specify the BigQuery dataset for the output table. This method is deprecated in favor of `ViewConfig.dataset`. ```javascript schema(schema: string): *this* ``` -------------------------------- ### columns Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets the column descriptors of columns in this view. Deprecated in favor of ViewConfig.columns. ```APIDOC ## columns ### Description Sets the column descriptors of columns in this view. ### Method ▸ columns(`columns`: ColumnDescriptor[]): *this* ### Parameters #### Path Parameters - `columns` (ColumnDescriptor[]) - Description not available ### Returns *this* ``` -------------------------------- ### Assertion SQL Query in Separate File Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md The SQL query for an assertion declared in an action config file, stored in a separate .sql file. ```sql -- definitions/name.sql SELECT * FROM table WHERE a IS NULL ``` -------------------------------- ### queries Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets the query/queries to generate the operation from. Deprecated in favor of a single `query(` method. ```APIDOC ## queries ### Description Sets the query/queries to generate the operation from. ### Method queries ### Parameters #### Path Parameters - **queries** ([Contextable]‹[IActionContext], string | string[]›) - Required - The query or queries to set. ### Returns *this* ``` -------------------------------- ### columns Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets the column descriptors of columns in this table. Deprecated in favor of OperationConfig.columns. ```APIDOC ## columns ### Description Sets the column descriptors of columns in this table. ### Method columns ### Parameters #### Path Parameters - **columns** (ColumnDescriptor[]) - Required - Description of the columns. ### Returns *this* ``` -------------------------------- ### Declare a Dataform Dataset Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Declares a dataset as a Dataform data source. Available only in the `/definitions` directory. ```javascript dataform.declare({ type: "table", name: "my_dataset", description: "My dataset description" }) ``` -------------------------------- ### columns Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets the column descriptors of columns in this table. Deprecated in favor of TableConfig.columns. ```APIDOC ## columns ### Description Sets the column descriptors of columns in this table. ### Method columns ### Parameters #### Path Parameters - **columns** (ColumnDescriptor[]) - Required - An array of column descriptors. ### Returns *this* ``` -------------------------------- ### Setting Post-Operations for a Table Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Use postOps to define operations that run after a table's query has been executed. This is useful for managing permissions, such as revoking temporary access. ```javascript // definitions/file.js publish("example") .preOps(ctx => `GRANT lexible_roles/bigquery.dataViewer lexible_roles ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE lexible_roles/bigquery.dataViewer lexible_roles ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) ``` -------------------------------- ### type Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets the type of the action. Deprecated in favor of setting the action type in the configs passed to action constructor functions. ```APIDOC ## type ### Description Sets the type of the action. Deprecated in favor of action type can being set in the configs passed to action constructor functions, for example `publish("name", { type: "table" })`. ### Method This is a method signature, not an HTTP endpoint. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None provided. ### Response #### Success Response * this - Returns the current instance for chaining. #### Response Example None provided. ``` -------------------------------- ### Set Assertion Dependencies using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md The 'dependencies' method sets the dependencies for the assertion. This is deprecated in favor of AssertionConfig.dependencies. ```javascript dependencies(value: Resolvable | Resolvable[]): this ``` -------------------------------- ### tags Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Sets user-defined tags for this action. Deprecated in favor of ViewConfig.tags. ```APIDOC ## tags ### Description Sets a list of user-defined tags applied to this action. ### Parameters - `value` (string | string[]) - A single tag or an array of tags. ### Returns *this* ### Deprecated Deprecated in favor of [ViewConfig.tags](configs#dataform-ActionConfig-ViewConfig). ``` -------------------------------- ### description Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets the description of this incremental table. Deprecated in favor of IncrementalTableConfig.description. ```APIDOC ## description ### Description Sets the description of this incremental table. ### Method ```typescript description(description: string): this ``` ### Parameters * `description` (string) - Required - The description for the incremental table. ### Returns * `this` ``` -------------------------------- ### Session.assert Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/session.md Adds a Dataform assertion to the compiled graph. Available only in the `/definitions` directory. ```APIDOC ## Session.assert ### Description Adds a Dataform assertion to the compiled graph. Available only in the `/definitions` directory. ### Method `assert(name: string, queryOrConfig?: AContextable | AssertionConfig): Assertion` ### Parameters #### Path Parameters - **name** (string) - Required - The name of the assertion. - **queryOrConfig?** (AContextable | AssertionConfig) - Optional - The query or configuration for the assertion. ### Returns - **Assertion** - The created Assertion object. ### See - [assertion](Assertion) for examples on how to use. ``` -------------------------------- ### Set BigQuery Options Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets BigQuery specific options for an action. This method is deprecated in favor of options available directly on `TableConfig`. ```javascript bigquery(bigquery: IBigQueryOptions): this ``` -------------------------------- ### Import NumPy Library Source: https://github.com/dataform-co/dataform/blob/main/examples/extreme_weather_programming/definitions/snowy_repository_creation.ipynb Imports the NumPy library, commonly used for numerical operations in Python. ```python import numpy as np ``` -------------------------------- ### Set Tags for a View (Deprecated) Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Use `tags` to apply user-defined tags to this action. This method is deprecated in favor of `ViewConfig.tags`. ```javascript tags(value: string | string[]): *this* ``` -------------------------------- ### Set Action Type (Deprecated) Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/view.md Use `type` to set the action type. This method is deprecated; action types should now be set in the configs passed to action constructor functions. ```javascript type(type: TableType): *this* ``` -------------------------------- ### Create Assertion using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md Utilize the Javascript API to programmatically create an assertion. The 'assert' function is used to name the assertion, and the 'query' method sets the test condition. ```javascript // definitions/file.js assert("name").query("SELECT * FROM table WHERE a IS NULL") ``` -------------------------------- ### Set Post-Operations for Table Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Sets a post-operation to run after the query is executed. This is often used for revoking temporary permissions. ```javascript postOps(posts: Contextable): this ``` ```javascript publish("example") .preOps(ctx => `GRANT `roles/bigquery.dataViewer` ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) .query(ctx => `SELECT * FROM ${ctx.ref("other_table")}`) .postOps(ctx => `REVOKE `roles/bigquery.dataViewer` ON TABLE ${ctx.ref("other_table")} TO "group:automation@example.com"`) ``` -------------------------------- ### Define Operation using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Create operations programmatically using the Javascript API. The 'operate' function initiates the operation, and the 'query' method sets the SQL statement. ```javascript // definitions/file.js operate("name").query("DELETE FROM dataset.table WHERE country = 'GB'") ``` -------------------------------- ### Set Assertion Database using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/assertion.md The 'database' method is used to set the Google Cloud project ID for the assertion. This is deprecated in favor of AssertionConfig.project. ```javascript database(database: string): this ``` -------------------------------- ### Define Incremental Table using Javascript API Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Use the Javascript API to publish an incremental table. The `when` function helps conditionally select rows based on whether the table is being incrementally updated. ```javascript publish("name", { type: "incremental" }).query( ctx => `SELECT ${ctx.when(ctx.incremental(), 1, 2) }` ) ``` -------------------------------- ### columns Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets the column descriptors of columns in this incremental table. Deprecated in favor of IncrementalTableConfig.columns. ```APIDOC ## columns ### Description Sets the column descriptors of columns in this incremental table. ### Method ```typescript columns(columns: ColumnDescriptor[]): this ``` ### Parameters * `columns` (ColumnDescriptor[]) - Required - An array of column descriptors. ### Returns * `this` ``` -------------------------------- ### hermetic Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md If true, this indicates that the action only depends on data from explicitly-declared dependencies. Deprecated in favor of TableConfig.hermetic. ```APIDOC ## hermetic ### Description If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. ### Method hermetic ### Parameters #### Path Parameters - **hermetic** (boolean) - Required - Indicates if the action is hermetic. ### Returns *void* ``` -------------------------------- ### tags Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets a list of user-defined tags applied to this action. Deprecated in favor of OperationConfig.tags. ```APIDOC ## tags ### Description Sets a list of user-defined tags applied to this action. ### Method tags ### Parameters #### Path Parameters - **value** (string | string[]) - Required - The tags to set. ### Returns *this* ``` -------------------------------- ### tags Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/incrementaltable.md Sets a list of user-defined tags applied to this action. Deprecated in favor of IncrementalTableConfig.tags. ```APIDOC ## tags ### Description Sets a list of user-defined tags applied to this action. Deprecated in favor of IncrementalTableConfig.tags. ### Method N/A ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body - **value** (string | string[]) - Required - The tags to apply to the action. ### Request Example N/A ### Response #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### description Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/operation.md Sets the description of this assertion. Deprecated in favor of OperationConfig.description. ```APIDOC ## description ### Description Sets the description of this assertion. ### Method description ### Parameters #### Path Parameters - **description** (string) - Required - The description for the assertion. ### Returns *this* ``` -------------------------------- ### Set Table Type (Deprecated) Source: https://github.com/dataform-co/dataform/blob/main/docs/reference/table.md Deprecated. Action type can now be set in configs passed to action constructor functions, e.g., publish("name", { type: "table" }). ```javascript type("table") ```