=============== LIBRARY RULES =============== From library maintainers: - test ### Install Go Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install the Google Cloud BigQuery client library for Go. ```go go get cloud.google.com/go/bigquery ``` -------------------------------- ### Install Node.js Authentication Libraries Source: https://docs.cloud.google.com/bigquery/docs/authentication/end-user-installed Install the google-auth-library and readline-promise npm packages for Node.js authentication. ```bash npm install google-auth-library npm install readline-promise ``` -------------------------------- ### C# BigQuery Client Library Setup Source: https://docs.cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv Demonstrates the initial setup for using the BigQuery C# client library, including authentication prerequisites. ```csharp using Google.Cloud.BigQuery.V2; using System; ``` -------------------------------- ### Install PHP Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install the Google Cloud BigQuery Analytics Hub client library for PHP using Composer. ```php composer require google/cloud-bigquery-analyticshub ``` -------------------------------- ### Install Ruby Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install the Google Cloud BigQuery Analytics Hub V1 client library for Ruby using gem. ```ruby gem install google-cloud-bigquery-analytics_hub-v1 ``` -------------------------------- ### Install C# Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install the Google Cloud BigQuery Analytics Hub V1 client library for C# using NuGet. ```csharp Install-Package Google.Cloud.BigQuery.AnalyticsHub.V1 -Pre ``` -------------------------------- ### Install Node.js Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install the Google Cloud BigQuery Data Exchange client library for Node.js using npm. ```javascript npm install @google-cloud/bigquery-data-exchange ``` -------------------------------- ### Example: Create a View with Differential Privacy Policy Source: https://docs.cloud.google.com/bigquery/docs/analysis-rules This example demonstrates creating a view named 'ExamView' with specific differential privacy settings. Note that delta and epsilon parameters are set high for this example due to a small dataset; in practice, they should be much smaller. ```sql CREATE OR REPLACE VIEW mydataset.ExamView OPTIONS( privacy_policy= '{"differential_privacy_policy": {"privacy_unit_column": "last_name", "max_epsilon_per_query": 1000.0, "epsilon_budget": 10000.1, "delta_per_query": 0.01, "delta_budget": 0.1, "max_groups_contributed": 2}}' ) AS ( SELECT * FROM mydataset.ExamTable ); -- NOTE: Delta and epsilon parameters are set very high due to the small -- dataset. In practice, these should be much smaller. ``` -------------------------------- ### Example: Create View with Aggregation Threshold Source: https://docs.cloud.google.com/bigquery/docs/analysis-rules This example demonstrates creating a view with a specific aggregation threshold and privacy unit column. ```sql CREATE OR REPLACE VIEW mydataset.ExamView OPTIONS( privacy_policy= '{"aggregation_threshold_policy": {"threshold": 3, "privacy_unit_column": "last_name"}}' ) AS ( SELECT * FROM mydataset.ExamTable ); ``` -------------------------------- ### Install Python Client Library Source: https://docs.cloud.google.com/bigquery/docs/reference/analytics-hub Install or upgrade the Google Cloud BigQuery Analytics Hub client library for Python using pip. ```python pip install --upgrade google-cloud-bigquery-analyticshub ``` -------------------------------- ### Install Python Authentication Library Source: https://docs.cloud.google.com/bigquery/docs/authentication/end-user-installed Install the oauthlib integration for Google Auth using pip. This is required for authenticating Python applications. ```bash pip install --upgrade google-auth-oauthlib ``` -------------------------------- ### Create View with List Overlap Rule Example Source: https://docs.cloud.google.com/bigquery/docs/analysis-rules Example of creating a view with a list overlap rule, specifying 'JOIN_ANY' condition and allowed columns. ```sql -- Create a view that includes a table called ExamTable. CREATE OR REPLACE VIEW mydataset.ExamView OPTIONS( privacy_policy= '{"join_restriction_policy": {"join_condition": "JOIN_ANY", "join_allowed_columns": ["test_id", "test_score"]}}' ) AS ( SELECT * FROM mydataset.ExamTable ); ``` -------------------------------- ### INSTR Function Example Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Demonstrates the usage of the INSTR function to find the starting position of the second occurrence of a substring within a string. The function returns the starting position (1-based index) of the substring. ```sql INSTR('helloooo', 'oo', 1, 2) AS instr; ``` -------------------------------- ### Select all files with a prefix in Cloud Storage Source: https://docs.cloud.google.com/bigquery/docs/batch-loading-data Use an asterisk wildcard to select all files in folders that start with a specific prefix. This example selects files from `gs://mybucket/fed-samples/fed-sample` and any subsequent characters. ```text gs://mybucket/fed-samples/fed-sample* ``` -------------------------------- ### Create Compute Engine Instance with Service Account Source: https://docs.cloud.google.com/bigquery/docs/authentication Create a Compute Engine instance and attach a service account to it. Replace INSTANCE_NAME, ZONE, and SERVICE_ACCOUNT_EMAIL with your specific values. ```bash gcloud compute instances create INSTANCE_NAME --zone=ZONE --service-account=SERVICE_ACCOUNT_EMAIL ``` -------------------------------- ### Get Table Information in Java Source: https://docs.cloud.google.com/bigquery/docs/tables This Java code snippet demonstrates how to retrieve table information, including its description, using the BigQuery client library. It requires proper setup of Application Default Credentials. ```java import com.google.cloud.bigquery.BigQuery; import com.google.cloud.bigquery.BigQueryException; import com.google.cloud.bigquery.BigQueryOptions; import com.google.cloud.bigquery.Table; import com.google.cloud.bigquery.TableId; public class GetTable { public static void runGetTable() { // TODO(developer): Replace these variables before running the sample. String projectId = "bigquery_public_data"; String datasetName = "samples"; String tableName = "shakespeare"; getTable(projectId, datasetName, tableName); } public static void getTable(String projectId, String datasetName, String tableName) { try { // Initialize client that will be used to send requests. This client only needs to be created // once, and can be reused for multiple requests. BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); TableId tableId = TableId.of(projectId, datasetName, tableName); Table table = bigquery.getTable(tableId); System.out.println("Table info: " + table.getDescription()); } catch (BigQueryException e) { System.out.println("Table not retrieved. \n" + e.toString()); } } } ``` -------------------------------- ### Get IAM policy for a reservation (Python) Source: https://docs.cloud.google.com/bigquery/docs/samples/bigqueryreservation-reservationservice-iampolicy-get Use this Python snippet to retrieve the IAM policy for a BigQuery reservation. Ensure you have followed the Python setup instructions and set up Application Default Credentials for authentication. ```python from google.api_core.exceptions import NotFound from google.cloud import bigquery_reservation_v1 client = bigquery_reservation_v1.ReservationServiceClient() def get_reservation_iam_policy(project_id: str, location: str, reservation_id: str): """Gets the IAM policy for a reservation. An IAM policy is a collection of bindings that associates one or more principals with a single role. This sample demonstrates how to retrieve the policy for a reservation. Args: project_id: The Google Cloud project ID. location: The geographic location of the reservation, such as "us-central1". reservation_id: The ID of the reservation to get the policy for. """ resource = client.reservation_path(project_id, location, reservation_id) try: policy = client.get_iam_policy(resource=resource) print(f"Got IAM policy for reservation: {resource}") for binding in policy.bindings: print(f" Role: {binding.role}") print(f" Members: {binding.members}") except NotFound: print(f"Reservation not found: {resource}") ``` -------------------------------- ### Get IAM policy for a reservation (Node.js) Source: https://docs.cloud.google.com/bigquery/docs/samples/bigqueryreservation-reservationservice-iampolicy-get Use this Node.js snippet to retrieve the IAM policy for a BigQuery reservation. Ensure you have followed the Node.js setup instructions and set up Application Default Credentials for authentication. ```javascript const { ReservationServiceClient, } = require('@google-cloud/bigquery-reservation').v1; const {status} = require('@grpc/grpc-js'); const client = new ReservationServiceClient(); /** * Gets the IAM policy for a reservation. * An IAM policy is a collection of bindings that associates one or more members, * such as service accounts or users, with a single role. * * @param {string} projectId - Google Cloud project ID, for example "example-project-id". * @param {string} location - Location of the reservation, for example "us-central1". * @param {string} reservationId - ID of the reservation, for example "example-reservation". */ async function getReservationIamPolicy( projectId, location = 'us-central1', reservationId = 'example-reservation', ) { const resource = `projects/${projectId}/locations/${location}/reservations/${reservationId}`; const request = { resource, }; try { const [policy] = await client.getIamPolicy(request); console.log(`Policy for reservation ${reservationId}:`); if (policy.bindings && policy.bindings.length > 0) { console.log(JSON.stringify(policy.bindings, null, 2)); } else { console.log('This reservation has no policy bindings.'); } } catch (err) { if (err.code === status.NOT_FOUND) { console.log( `Reservation '${reservationId}' not found in project '${projectId}' at location '${location}'.`, ); } else { console.error('Error getting IAM policy:', err); } } } ``` -------------------------------- ### Avro Schema for RANGE Type Source: https://docs.cloud.google.com/bigquery/docs/reference/storage?hl=en Example Avro schema definition for a RANGE type, demonstrating how to represent start and end boundaries with logical types. This shows both a full record definition and a reference to a previously defined record. ```json { "name": "range_date_1", "type": { "type": "record", "namespace": "google.sqlType", "name": "RANGE_DATE", "sqlType": "RANGE", "fields": [ { "name": "start", "type": ["null", {"type": "int", "logicalType": "date"}] }, { "name": "end", "type": ["null", {"type": "int", "logicalType": "date"}] }, ] } }, { "name": "range_date_2", "type": "google.sqlType.RANGE_DATE" } ``` -------------------------------- ### Run Query and Get Total Rows in Java Source: https://docs.cloud.google.com/bigquery/docs/samples/bigquery-query-total-rows Use this Java code to execute a SQL query against BigQuery and print the total number of rows returned. Ensure you have followed the Java setup instructions and set up Application Default Credentials. ```Java import com.google.cloud.bigquery.BigQuery; import com.google.cloud.bigquery.BigQueryException; import com.google.cloud.bigquery.BigQueryOptions; import com.google.cloud.bigquery.QueryJobConfiguration; import com.google.cloud.bigquery.TableResult; // Sample to run query total rows public class QueryTotalRows { public static void main(String[] args) { String query = "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013`" + " WHERE state = \"TX\"" + " LIMIT 100"; queryTotalRows(query); } public static void queryTotalRows(String query) { try { // Initialize client that will be used to send requests. This client only needs to be created // once, and can be reused for multiple requests. BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); TableResult results = bigquery.query(QueryJobConfiguration.of(query)); System.out.println("Query total rows performed successfully." + results.getTotalRows()); } catch (BigQueryException | InterruptedException e) { System.out.println("Query not performed \n" + e.toString()); } } } ``` -------------------------------- ### Create BigQuery Dataset using Go with ADC Source: https://docs.cloud.google.com/bigquery/docs/authentication/getting-started This Go code sample demonstrates how to create a BigQuery dataset. It relies on Application Default Credentials for authentication. ```go // Sample bigquery-quickstart creates a Google BigQuery dataset. package main import ( "context" "fmt" "log" "cloud.google.com/go/bigquery" ) func main() { ctx := context.Background() // Sets your Google Cloud Platform project ID. projectID := "YOUR_PROJECT_ID" // Creates a client. client, err := bigquery.NewClient(ctx, projectID) if err != nil { log.Fatalf("bigquery.NewClient: %v", err) } defer client.Close() // Sets the name for the new dataset. datasetName := "my_new_dataset" // Creates the new BigQuery dataset. if err := client.Dataset(datasetName).Create(ctx, &bigquery.DatasetMetadata{}); err != nil { log.Fatalf("Failed to create dataset: %v", err) } fmt.Printf("Dataset created\n") } ``` -------------------------------- ### Select specific CSV files in a Cloud Storage folder Source: https://docs.cloud.google.com/bigquery/docs/batch-loading-data Use an asterisk wildcard to select files matching a specific naming pattern with a `.csv` extension within a single folder. This example selects files starting with `fed-sample` and ending with `.csv` in `gs://mybucket/fed-samples/`, but not in subfolders. ```text gs://mybucket/fed-samples/fed-sample*.csv ``` -------------------------------- ### Initialize gcloud CLI Source: https://docs.cloud.google.com/bigquery/docs/analytics-hub-custom-constraints Run this command to initialize the gcloud command-line interface. Ensure you are signed in with your Google Cloud account and have a project selected. ```bash gcloud init ``` -------------------------------- ### Run Query and Get Total Rows in Python Source: https://docs.cloud.google.com/bigquery/docs/samples/bigquery-query-total-rows This Python code snippet demonstrates how to run a BigQuery query and retrieve the total number of rows. It requires Python client library setup and Application Default Credentials. The query location must match the dataset location. ```Python # from google.cloud import bigquery # client = bigquery.Client() query = ( "SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` " 'WHERE state = "TX" ' "LIMIT 100" ) results = client.query_and_wait( query, # Location must match that of the dataset(s) referenced in the query. location="US", ) # API request - starts the query and waits for results. print("Got {} rows.".format(results.total_rows)) ``` -------------------------------- ### Get character length of a string Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Gets the number of characters in a STRING value. ```sql SELECT CHAR_LENGTH('абвгд') AS char_length; /*-------------+-------------+ | char_length | +-------------+-------------+ | 5 | +------------ */ ``` -------------------------------- ### BigQuery Python Client Library Setup Source: https://docs.cloud.google.com/bigquery/docs/access-historical-data This snippet shows the necessary import for using the BigQuery Python client library. Ensure you have followed the Python setup instructions and configured authentication. ```python import time from google.cloud import bigquery ``` -------------------------------- ### Initialize BigQuery client (Python) Source: https://docs.cloud.google.com/bigquery/docs/adding-labels Initializes the BigQuery client for use in Python samples. Ensure you have set up Application Default Credentials. ```python from google.cloud import bigquery client = bigquery.Client() ``` -------------------------------- ### Create BigQuery Table with Explicit Schema (Go) Source: https://docs.cloud.google.com/bigquery/docs/tables Demonstrates creating a new BigQuery table and specifying a schema using the Go client library. Ensure Application Default Credentials are set up. ```go import ( "context" "fmt" "time" "cloud.google.com/go/bigquery" ) // createTableExplicitSchema demonstrates creating a new BigQuery table and specifying a schema. func createTableExplicitSchema(projectID, datasetID, tableID string) error { // projectID := "my-project-id" // datasetID := "mydatasetid" // tableID := "mytableid" ctx := context.Background() client, err := bigquery.NewClient(ctx, projectID) if err != nil { return fmt.Errorf("bigquery.NewClient: %v", err) } defer client.Close() sampleSchema := bigquery.Schema{ {Name: "full_name", Type: bigquery.StringFieldType}, {Name: "age", Type: bigquery.IntegerFieldType}, } metaData := &bigquery.TableMetadata{ Schema: sampleSchema, ExpirationTime: time.Now().AddDate(1, 0, 0), // Table will be automatically deleted in 1 year. } tableRef := client.Dataset(datasetID).Table(tableID) if err := tableRef.Create(ctx, metaData); err != nil { return err } return nil } ``` -------------------------------- ### Search Statistics Example Source: https://docs.cloud.google.com/bigquery/docs/search An example of the `searchStatistics` field in job details, indicating `indexUsageMode` and `indexUnusedReasons`. ```json "searchStatistics": { "indexUnusedReasons": [ { "baseTable": { "datasetId": "my_dataset", "projectId": "my_project", "tableId": "my_table" }, "code": "INDEX_CONFIG_NOT_AVAILABLE", "message": "There is no search index configuration for the base table `my_project:my_dataset.my_table`." } ], "indexUsageMode": "UNUSED" }, ``` -------------------------------- ### Create Search Index with Custom LOG_ANALYZER Options Source: https://docs.cloud.google.com/bigquery/docs/search This example demonstrates creating a search index on a table, specifying the LOG_ANALYZER with custom token filter options defined in a JSON string. ```sql CREATE TABLE dataset.complex_table( a STRING, my_struct STRUCT, b ARRAY ); CREATE SEARCH INDEX my_index ON dataset.complex_table(a, my_struct, b) OPTIONS (analyzer = 'LOG_ANALYZER', analyzer_options = '''{ "token_filters": [ { "normalizer": {"mode": "NONE"} } ] }'''); ``` -------------------------------- ### Get character length using CHARACTER_LENGTH Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Synonym for CHAR_LENGTH. Gets the number of characters in a STRING value. ```sql SELECT 'абвгд' AS characters, CHARACTER_LENGTH('абвгд') AS char_length_example /*------------+---------------------+ | characters | char_length_example | +------------+---------------------+ | абвгд | 5 | +------------+---------------------*/ ``` -------------------------------- ### Find position of a regex match in a string Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Use REGEXP_INSTR to find the starting position of a regular expression match within a source string. It supports specifying the starting position, occurrence number, and whether to return the start or end+1 of the match. ```SQL SELECT REGEXP_INSTR('ab@cd-ef', '@[^-]*') AS instr_a, REGEXP_INSTR('ab@d-ef', '@[^-]*') AS instr_b, REGEXP_INSTR('abc@cd-ef', '@[^-]*') AS instr_c, REGEXP_INSTR('abc-ef', '@[^-]*') AS instr_d /*---------------------------------------+ | instr_a | instr_b | instr_c | instr_d | +---------------------------------------+ | 3 | 3 | 4 | 0 | +---------------------------------------*/ ``` ```SQL SELECT REGEXP_INSTR('a@cd-ef b@cd-ef', '@[^-]*', 1) AS instr_a, REGEXP_INSTR('a@cd-ef b@cd-ef', '@[^-]*', 2) AS instr_b, REGEXP_INSTR('a@cd-ef b@cd-ef', '@[^-]*', 3) AS instr_c, REGEXP_INSTR('a@cd-ef b@cd-ef', '@[^-]*', 4) AS instr_d /*---------------------------------------+ | instr_a | instr_b | instr_c | instr_d | +---------------------------------------+ | 2 | 2 | 10 | 10 | +---------------------------------------*/ ``` ```SQL SELECT REGEXP_INSTR('a@cd-ef b@cd-ef c@cd-ef', '@[^-]*', 1, 1) AS instr_a, REGEXP_INSTR('a@cd-ef b@cd-ef c@cd-ef', '@[^-]*', 1, 2) AS instr_b, REGEXP_INSTR('a@cd-ef b@cd-ef c@cd-ef', '@[^-]*', 1, 3) AS instr_c /*-----------------------------+ | instr_a | instr_b | instr_c | +-----------------------------+ | 2 | 10 | 18 | +-----------------------------*/ ``` ```SQL SELECT REGEXP_INSTR('a@cd-ef', '@[^-]*', 1, 1, 0) AS instr_a, REGEXP_INSTR('a@cd-ef', '@[^-]*', 1, 1, 1) AS instr_b /*-------------------+ | instr_a | instr_b | +-------------------+ | 2 | 5 | +-------------------*/ ``` -------------------------------- ### Create Terraform Directory and Main File Source: https://docs.cloud.google.com/bigquery/docs/annotate-image Create a new directory for your Terraform configuration and initialize the main.tf file within it. ```bash mkdir DIRECTORY && cd DIRECTORY && touch main.tf ``` -------------------------------- ### Invalid FORMAT string example with '%' Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions An example of an invalid FORMAT string expression due to an incomplete format specifier. ```sql SELECT FORMAT('%'); -- Error: Invalid format string ``` -------------------------------- ### Invalid FORMAT string example Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions An example of an invalid FORMAT string expression due to an incorrect number of arguments. ```sql SELECT FORMAT('%s', 1); -- Error: Argument of type INT64 is not convertible to STRING ``` -------------------------------- ### SEARCH Function Examples with LOG_ANALYZER Source: https://docs.cloud.google.com/bigquery/docs/search Illustrates various calls to the SEARCH function using the default LOG_ANALYZER, showcasing different search terms and expected boolean results based on tokenization and matching rules. ```sql SEARCH('foobarexample', NULL) ``` ```sql SEARCH('foobarexample', '') ``` ```sql SEARCH('foobar-example', 'foobar example') ``` ```sql SEARCH('foobar-example', 'foobarexample') ``` ```sql SEARCH('foobar-example', 'foobar\&example') ``` ```sql SEARCH('foobar-example', R'foobar&example') ``` ```sql SEARCH('foobar-example', '`foobar&example`') ``` ```sql SEARCH('foobar&example', '`foobar&example`') ``` ```sql SEARCH('foobar-example', 'example foobar') ``` ```sql SEARCH('foobar-example', 'foobar example') ``` ```sql SEARCH('foobar-example', '`foobar-example`') ``` ```sql SEARCH('foobar-example', '`foobar`') ``` ```sql SEARCH('`foobar-example`', '`foobar-example`') ``` ```sql SEARCH('foobar@example.com', '`example.com`') ``` ```sql SEARCH('a foobar-example b', '`foobar-example`') ``` ```sql SEARCH(['foobar', 'example'], 'foobar example') ``` ```sql SEARCH('foobar=', '`foobar\=') ``` ```sql SEARCH('foobar=', R'`foobar\=`') ``` ```sql SEARCH('foobar=', 'foobar\=') ``` ```sql SEARCH('foobar=', R'foobar\=') ``` ```sql SEARCH('foobar.example', '`foobar`') ``` ```sql SEARCH('foobar.example', '`foobar.`') ``` ```sql SEARCH('foobar..example', '`foobar.`') ``` -------------------------------- ### Extract substring when start position is out of bounds Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Returns an empty substring when the specified start position is beyond the length of the string. ```sql SELECT SUBSTR('apple', 123) AS example /*---------+ | example | +---------+ | | +---------*/ ``` -------------------------------- ### Find Substring Starting from Specific Position Source: https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/string_functions Finds the first occurrence of 'an' in 'banana', starting the search from the 3rd character. ```sql SELECT 'banana' AS value, 'an' AS subvalue, 3 AS position, 1 AS occurrence, INSTR('banana', 'an', 3, 1) AS instr; /*--------------+--------------+----------+------------+-------+ | value | subvalue | position | occurrence | instr | +--------------+--------------+----------+------------+-------+ | banana | an | 3 | 1 | 4 | +--------------+--------------+----------+------------+-------*/ ``` -------------------------------- ### Create Listing API Endpoint Source: https://docs.cloud.google.com/bigquery/docs/analytics-hub-manage-listings Use the `projects.locations.dataExchanges.listings.create` method to create a new listing. Replace the placeholder values with your specific project, location, data exchange, and listing IDs. ```bash POST https://analyticshub.googleapis.com/v1/projects/PROJECT_ID/location/LOCATION/dataExchanges/DATAEXCHANGE_ID/listings?listingId=LISTING_ID ``` -------------------------------- ### Initialize BigQuery Client using Python with ADC Source: https://docs.cloud.google.com/bigquery/docs/authentication/getting-started This Python snippet shows how to initialize the BigQuery client. If credentials are not explicitly provided, it defaults to using Application Default Credentials. ```python from google.cloud import bigquery # If you don't specify credentials when constructing the client, ``` -------------------------------- ### Initialize BigQuery DataFrames and Load Data Source: https://docs.cloud.google.com/bigquery/docs/dataframes-quickstart Sets up BigQuery DataFrames options, including project ID and ordering mode, then reads data from a BigQuery table into a DataFrame and previews the results. ```python import bigframes.pandas as bpd # Set BigQuery DataFrames options # Note: The project option is not required in all environments. # On BigQuery Studio, the project ID is automatically detected. bpd.options.bigquery.project = your_gcp_project_id # Use "partial" ordering mode to generate more efficient queries, but the # order of the rows in DataFrames may not be deterministic if you have not # explictly sorted it. Some operations that depend on the order, such as # head() will not function until you explictly order the DataFrame. Set the # ordering mode to "strict" (default) for more pandas compatibility. bpd.options.bigquery.ordering_mode = "partial" # Create a DataFrame from a BigQuery table query_or_table = "bigquery-public-data.ml_datasets.penguins" df = bpd.read_gbq(query_or_table) # Efficiently preview the results using the .peek() method. df.peek() ``` -------------------------------- ### 使用 Go 客户端库执行试运行查询 Source: https://docs.cloud.google.com/bigquery/docs/best-practices-costs?hl=zh-cn 通过将 `DryRun` 字段设置为 `true` 来执行试运行查询,以验证查询结构并估算扫描的字节数。此方法适用于在实际执行前检查查询。 ```go import ( "context" "fmt" "io" "cloud.google.com/go/bigquery" ) // queryDryRun demonstrates issuing a dry run query to validate query structure and // provide an estimate of the bytes scanned. func queryDryRun(w io.Writer, projectID string) error { // projectID := "my-project-id" ctx := context.Background() client, err := bigquery.NewClient(ctx, projectID) if err != nil { return fmt.Errorf("bigquery.NewClient: %v", err) } defer client.Close() q := client.Query(` SELECT name, COUNT(*) as name_count FROM ` + "`bigquery-public-data.usa_names.usa_1910_2013`" + ` WHERE state = 'WA' GROUP BY name`) q.DryRun = true // Location must match that of the dataset(s) referenced in the query. q.Location = "US" job, err := q.Run(ctx) if err != nil { return err } // Dry run is not asynchronous, so get the latest status and statistics. status := job.LastStatus() if err := status.Err(); err != nil { return err } fmt.Fprintf(w, "This query will process %d bytes\n", status.Statistics.TotalBytesProcessed) return nil } ``` -------------------------------- ### Query Job Performance Over Five Days Source: https://docs.cloud.google.com/bigquery/docs/admin-resource-charts This example demonstrates how to query job details over a specific five-day period. It uses a concrete project ID, region, and time range. ```sql SELECT AVG(TIMESTAMP_DIFF(end_time, creation_time, MILLISECOND)) as avg_latency_ms, SUM(total_bytes_processed) as total_bytes, COUNT(*) as query_numbers, FROM `myproject.region-us`.INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION WHERE creation_time >= '2024-06-25 00:00:00-07' AND creation_time < '2024-06-30 00:00:00-07' AND (statement_type != "SCRIPT" OR statement_type IS NULL) AND reservation_id = reservationID ``` -------------------------------- ### YAML IAM Policy Example Source: https://docs.cloud.google.com/bigquery/docs/reference/bigqueryconnection/rest/Shared.Types/Policy A YAML representation of an IAM policy, illustrating the same structure as the JSON example with roles, members, and conditions. ```yaml bindings: - members: - user:mike@example.com - group:admins@example.com - domain:google.com - serviceAccount:my-project-id@appspot.gserviceaccount.com role: roles/resourcemanager.organizationAdmin - members: - user:eve@example.com role: roles/resourcemanager.organizationViewer condition: title: expirable access description: Does not grant access after Sep 2020 expression: request.time < timestamp('2020-10-01T00:00:00.000Z') etag: BwWWja0YfJA= version: 3 ```