### Installation Source: https://context7.com/uber/athenadriver/llms.txt Install the Athena Driver using the go get command. ```APIDOC ## Installation ```go go install github.com/uber/athenadriver@latest ``` ``` -------------------------------- ### Install Dependencies Source: https://github.com/uber/athenadriver/blob/master/resources/CONTRIBUTING.md Install the project's dependencies using the provided Makefile. ```bash make dependencies ``` -------------------------------- ### Build Simple Query Example Source: https://github.com/uber/athenadriver/blob/master/README.md Build the Go program that executes a simple Athena query. ```bash $ go build examples/query/dml_select_simple.go ``` -------------------------------- ### Install athenareader using Go Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Use this command to install the athenareader utility tool. Ensure you have Go installed and configured. ```shell go get -u github.com/uber/athenadriver/athenareader ``` -------------------------------- ### Build Integration Test Example Source: https://github.com/uber/athenadriver/blob/master/README.md Build the concurrency test executable located in the examples/perf directory. ```bash $cd $GOPATH/src/github.com/uber/athenadriver $go build examples/perf/concurrency.go ``` -------------------------------- ### Enable AWS Profile Manual Setup for Authentication Source: https://github.com/uber/athenadriver/blob/master/ChangeLog.txt Sample code demonstrating how to enable manual AWS profile setup for authentication with AthenaDriver. Refer to the linked documentation for more details. ```Go package main import ( "context" "fmt" "log" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/athena" "github.com/uber/athenadriver/athenadriver" ) func main() { // Use the AWS SDK's default credential chain, which includes looking for credentials in the shared credentials file (~/.aws/credentials) or environment variables. // You can specify a profile name using the "AWS_PROFILE" environment variable. cfg, err := config.LoadDefaultConfig(context.TODO()) if err != nil { log.Fatalf("unable to load SDK config, %v", err) } // Create an Athena client from the loaded configuration. client := athena.NewFromConfig(cfg) // Initialize the AthenaDriver with the Athena client. driver, err := athenadriver.NewDriver(client) if err != nil { log.Fatalf("unable to create driver, %v", err) } // Example query. query := "SELECT 1" // Execute the query. rows, err := driver.Query(context.TODO(), query) if err != nil { log.Fatalf("unable to query, %v", err) } // Process the results. for rows.Next() { var result int if err := rows.Scan(&result); err != nil { log.Fatalf("unable to scan row, %v", err) } fmt.Printf("Result: %d\n", result) } if err := rows.Err(); err != nil { log.Fatalf("error iterating rows, %v", err) } } ``` -------------------------------- ### Install Athena Driver Source: https://context7.com/uber/athenadriver/llms.txt Use the go install command to add the driver to your project. ```bash go install github.com/uber/athenadriver@latest ``` -------------------------------- ### Install athenadriver (Go < 1.17) Source: https://github.com/uber/athenadriver/blob/master/README.md Use 'go get' to install the athenadriver package for Go versions prior to 1.17. ```go go get -u github.com/uber/athenadriver ``` -------------------------------- ### Install AWS Lambda Go dependency Source: https://github.com/uber/athenadriver/blob/master/examples/lambda/Go/README.md Use this command to fetch the required AWS Lambda library for Go. ```bash go get github.com/aws/aws-lambda-go/lambda ``` -------------------------------- ### Run Simple Query Example Source: https://github.com/uber/athenadriver/blob/master/README.md Execute the compiled Go program to run a simple Athena query and display the result. ```bash $ ./dml_select_simple ``` -------------------------------- ### Clone Repository and Setup Environment Source: https://github.com/uber/athenadriver/blob/master/resources/CONTRIBUTING.md Clone the athenadriver repository and set up your local development environment by fetching upstream changes. ```bash mkdir -p $GOPATH/src/uber cd $GOPATH/src/uber git clone git@github.com:your_github_username/athenadriver.git cd athenadriver git remote add upstream https://github.com/uber/athenadriver.git git fetch upstream ``` -------------------------------- ### Simple Athena Query Example Source: https://github.com/uber/athenadriver/blob/master/README.md Demonstrates a basic query execution using athenadriver and the standard Go database/sql package. Replace placeholder AWS credentials and region with your actual values. ```go package main import ( "database/sql" drv "github.com/uber/athenadriver/go" ) func main() { // Step 1. Set AWS Credential in Driver Config. conf, _ := drv.NewDefaultConfig("s3://myqueryresults/", "us-east-2", "DummyAccessID", "DummySecretAccessKey") // Step 2. Open Connection. db, _ := sql.Open(drv.DriverName, conf.Stringify()) // Step 3. Query and print results var url string _ = db.QueryRow("SELECT url from sampledb.elb_logs limit 1").Scan(&url) println(url) } ``` -------------------------------- ### Query Athena with AthenaDriver in AWS Lambda Source: https://github.com/uber/athenadriver/blob/master/ChangeLog.txt Example demonstrating how to query Athena using AthenaDriver within an AWS Lambda function. This setup is useful for serverless data processing tasks. ```Go package main import ( "context" "fmt" "log" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/athena" "github.com/uber/athenadriver/athenadriver" ) func main() { // Load the default AWS configuration. In an AWS Lambda environment, this will typically use the Lambda execution role for credentials. cfg, err := config.LoadDefaultConfig(context.TODO()) if err != nil { log.Fatalf("unable to load SDK config, %v", err) } // Create an Athena client. client := athena.NewFromConfig(cfg) // Initialize the AthenaDriver. driver, err := athenadriver.NewDriver(client) if err != nil { log.Fatalf("unable to create driver, %v", err) } // Define your Athena query. query := "SELECT count(*) FROM \"your_database\Возможно, \"your_table\" // Execute the query. rows, err := driver.Query(context.TODO(), query) if err != nil { log.Fatalf("unable to query, %v", err) } // Process the query results. for rows.Next() { var count int if err := rows.Scan(&count); err != nil { log.Fatalf("unable to scan row, %v", err) } fmt.Printf("Query result count: %d\n", count) } if err := rows.Err(); err != nil { log.Fatalf("error iterating rows, %v", err) } } ``` -------------------------------- ### Configure Athena QueryExecutionContext Source: https://github.com/uber/athenadriver/blob/master/resources/UndocumentedAthena.md Sets the database and output location when starting an Athena query execution. ```go resp, err := c.athenaAPI.StartQueryExecution(&athena.StartQueryExecutionInput{ QueryString: aws.String(query), QueryExecutionContext: &athena.QueryExecutionContext{ Database: aws.String(c.connector.AthenaConfig.GetDB()), }, ResultConfiguration: &athena.ResultConfiguration{ OutputLocation: aws.String(c.connector.AthenaConfig.GetOutputBucket()), }, WorkGroup: aws.String(wg.Name), }) ``` -------------------------------- ### Create Table As Select Query Source: https://github.com/uber/athenadriver/blob/master/README.md Example CTAS SQL query that creates a new table from an existing one. ```sql CREATE TABLE sampledb.elb_logs_copy WITH ( format = 'TEXTFILE', external_location = 's3://external-location-test/elb_logs_copy', partitioned_by = ARRAY['ssl_protocol']) AS SELECT * FROM sampledb.elb_logs ``` -------------------------------- ### Get Driver Version using Pseudo Command Source: https://github.com/uber/athenadriver/blob/master/README.md Use the `pc:get_driver_version` pseudo command to retrieve the version of the athenadriver. ```go pc:get_driver_version ``` -------------------------------- ### Configure Logging and Metrics Source: https://context7.com/uber/athenadriver/llms.txt Integrate zap for logging and tally for metrics by passing them through the context. Requires enabling logging and metrics on the configuration object. ```go package main import ( "context" "database/sql" "time" "github.com/uber-go/tally/v4" drv "github.com/uber/athenadriver/go" "go.uber.org/zap" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") // Enable logging conf.SetLogging(true) // Enable metrics conf.SetMetrics(true) db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Create a zap logger logger, _ := zap.NewProduction() defer logger.Sync() // Create a tally scope for metrics scope, closer := tally.NewRootScope(tally.ScopeOptions{ Prefix: "athena_driver", Tags: map[string]string{"service": "my-app"}, }, time.Second) defer closer.Close() // Create context with logger and metrics ctx := context.Background() ctx = context.WithValue(ctx, drv.LoggerKey, logger) ctx = context.WithValue(ctx, drv.MetricsKey, scope) // Queries using this context will log and emit metrics rows, _ := db.QueryContext(ctx, "SELECT COUNT(*) FROM sampledb.elb_logs") defer rows.Close() // Metrics emitted include: // - awsathena.connector.connect (timer) // - awsathena.query.workgroup (timer) // - awsathena.query.startqueryexecution (timer) // - awsathena.query.queryexecutionstatesucceeded (timer) } ``` -------------------------------- ### Build and package Go Lambda function Source: https://github.com/uber/athenadriver/blob/master/examples/lambda/Go/README.md Initialize the module and compile the Go code into a zip archive for deployment. ```bash go mod init lambda-test go build main.go && zip function.zip main ``` -------------------------------- ### Create Driver Configuration with NewDefaultConfig Source: https://context7.com/uber/athenadriver/llms.txt Initializes the driver with explicit AWS credentials, region, and S3 output bucket. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { // Create configuration with S3 bucket, region, and AWS credentials conf, err := drv.NewDefaultConfig( "s3://my-query-results-bucket/", // S3 bucket for query results "us-east-2", // AWS region "AKIAIOSFODNN7EXAMPLE", // AWS Access Key ID "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", // Secret Access Key ) if err != nil { log.Fatal(err) } // Open database connection using the configuration db, err := sql.Open(drv.DriverName, conf.Stringify()) if err != nil { log.Fatal(err) } defer db.Close() // Execute a simple query var result int err = db.QueryRow("SELECT 123").Scan(&result) if err != nil { log.Fatal(err) } fmt.Println("Result:", result) // Output: Result: 123 } ``` -------------------------------- ### Execute a query from a local file Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Specify a local SQL file path using the -q flag to run complex queries. ```bash $ athenareader -d sampledb -b s3://example-athena-query-result -q tools/query.sql request_timestamp,elb_name 2015-01-06T00:00:00.516940Z,elb_demo_009 ``` -------------------------------- ### Get Query ID Status using Pseudo Command Source: https://github.com/uber/athenadriver/blob/master/README.md Use the `pc:get_query_id_status` pseudo command to retrieve the status of a given Query ID. ```go pc:get_query_id_status Query_ID ``` -------------------------------- ### Prepared Statement for Athena DB in Go Source: https://github.com/uber/athenadriver/blob/master/README.md Demonstrates how to use prepared statements with the athenadriver to handle user input securely in queries. This is useful when parts of the query come from external sources. ```go package main import ( "database/sql" drv "github.com/uber/athenadriver/go" ) func main() { // 1. Set AWS Credential in Driver Config. conf, _ := drv.NewDefaultConfig("s3://myqueryresults/", "us-east-2", "DummyAccessID", "DummySecretAccessKey") // 2. Open Connection. db, _ := sql.Open(drv.DriverName, conf.Stringify()) // 3. Prepared Statement statement, err := db.Prepare("CREATE TABLE sampledb.urls AS " + "SELECT url FROM sampledb.elb_logs where request_ip=? limit ?") if err != nil { panic(err) } // 4. Execute prepared Statement if result, e := statement.Exec("244.157.42.179", 2); e == nil { if rowsAffected, err := result.RowsAffected(); err == nil { println(rowsAffected) } } } ``` -------------------------------- ### Perform Cross-Database Join Source: https://github.com/uber/athenadriver/blob/master/resources/UndocumentedAthena.md Demonstrates creating tables in different databases and joining them using standard SQL syntax. ```sql -- prepare two demo tables in different databases CREATE TABLE default.elb_logs_new2 AS SELECT * FROM sampledb.elb_logs limit 1; CREATE TABLE sampledb.elb_logs_new1 AS SELECT * FROM sampledb.elb_logs limit 10; -- select-join query them SELECT a.elb_name, a.url FROM elb_logs_new2 a LEFT JOIN sampledb.elb_logs_new1 b ON a.request_timestamp=b.request_timestamp; ``` -------------------------------- ### Get Query ID using Pseudo Command Source: https://github.com/uber/athenadriver/blob/master/README.md Use the `pc:get_query_id` pseudo command to retrieve the Query ID of an SQL statement. This works regardless of whether the query succeeds or fails. ```go package main import ( "database/sql" "os" secret "github.com/uber/athenadriver/examples/constants" drv "github.com/uber/athenadriver/go" ) func main() { // 1. Set AWS Credential in Driver Config. os.Setenv("AWS_SDK_LOAD_CONFIG", "1") conf, err := drv.NewDefaultConfig(secret.OutputBucket, secret.Region, secret.AccessID, secret.SecretAccessKey) conf.SetLogging(true) if err != nil { panic(err) return } // 2. Open Connection. dsn := conf.Stringify() db, _ := sql.Open(drv.DriverName, dsn) // 3. Query with pseudo command `pc:get_query_id` var qid string _ = db.QueryRow("pc:get_query_id select url from sampledb.elb_logs limit 2").Scan(&qid) println("Query ID: ", qid) } ``` -------------------------------- ### Enable Cost Tracking with SetMoneyWise Source: https://context7.com/uber/athenadriver/llms.txt Activate cost tracking to output query costs in USD to stdout. This configuration must be set on the driver configuration object before opening the database connection. ```go package main import ( "database/sql" "fmt" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") // Enable cost tracking conf.SetMoneyWise(true) db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Each query will print cost information to stdout rows, _ := db.Query("SELECT * FROM sampledb.elb_logs LIMIT 1000") defer rows.Close() // Console output: // query cost: 0.00004768371582031250 USD, scanned data: 10485760 B, qid: abc123... var count int for rows.Next() { count++ } fmt.Printf("Retrieved %d rows\n", count) } ``` -------------------------------- ### Manage Asynchronous Queries with Pseudo Commands Source: https://context7.com/uber/athenadriver/llms.txt Use pc: prefixed commands to execute queries asynchronously, poll for status, and stop execution. Requires the athenadriver package. ```go package main import ( "database/sql" "fmt" "time" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Get driver version var version string db.QueryRow("pc:get_driver_version").Scan(&version) fmt.Println("Driver version:", version) // Output: Driver version: 1.1.15 // Get Query ID without waiting for results var queryID string db.QueryRow("pc:get_query_id SELECT COUNT(*) FROM sampledb.elb_logs").Scan(&queryID) fmt.Println("Query ID:", queryID) // Output: Query ID: a44f8e61-4cbb-429a-b7ab-bea2c4a5caed // Check query status var status string db.QueryRow("pc:get_query_id_status " + queryID).Scan(&status) fmt.Println("Status:", status) // Output: Status: RUNNING (or SUCCEEDED, FAILED, CANCELLED) // Poll until complete for status == "RUNNING" || status == "QUEUED" { time.Sleep(2 * time.Second) db.QueryRow("pc:get_query_id_status " + queryID).Scan(&status) fmt.Println("Status:", status) } // If query succeeded, fetch results using the Query ID directly if status == "SUCCEEDED" { rows, _ := db.Query(queryID) // Query using QID returns cached results defer rows.Close() var count int64 if rows.Next() { rows.Scan(&count) fmt.Printf("Count: %d\n", count) } } // Stop a running query var result string db.QueryRow("pc:stop_query_id " + queryID).Scan(&result) fmt.Println("Stop result:", result) // Output: Stop result: OK } ``` -------------------------------- ### Enable Zap Logging Source: https://github.com/uber/athenadriver/blob/master/README.md Pass a zap Production logger into the context to enable detailed driver logging. ```go package main import ( "context" "database/sql" "log" "time" "go.uber.org/zap" drv "github.com/uber/athenadriver/go" ) func main() { // 1. Set AWS Credential in Driver Config. conf, _ := drv.NewDefaultConfig("s3://query-results-bucket-test/", "us-east-2", "dummy-to-be-replaced", "dummy-to-be-replaced") // 2. Open Connection. dsn := conf.Stringify() db, _ := sql.Open(drv.DBDriverName, dsn) logger, _ := zap.NewProduction() defer logger.Sync() // 3. Query cancellation after 2 seconds ctx, _ := context.WithTimeout(context.Background(), 2*time.Second) ctx = context.WithValue(ctx, drv.LoggerKey, logger) rows, err := db.QueryContext(ctx, "select count(*) from sampledb.elb_logs") if err != nil { log.Fatal(err) return } defer rows.Close() } ``` -------------------------------- ### Use Prepared Statements with db.Prepare Source: https://context7.com/uber/athenadriver/llms.txt Implements prepared statements by interpolating parameters safely, as Athena does not support them natively. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Prepare a statement with placeholders stmt, err := db.Prepare(` SELECT url, request_timestamp FROM sampledb.elb_logs WHERE request_ip = ? LIMIT ? `) if err != nil { log.Fatal(err) } defer stmt.Close() // Execute with different parameters rows, err := stmt.Query("244.157.42.179", 5) if err != nil { log.Fatal(err) } defer rows.Close() for rows.Next() { var url, timestamp string rows.Scan(&url, ×tamp) fmt.Printf("URL: %s, Time: %s\n", url, timestamp) } // Execute for side effects (CTAS with prepared statement) createStmt, _ := db.Prepare(` CREATE TABLE sampledb.filtered_logs AS SELECT * FROM sampledb.elb_logs WHERE request_ip = ? LIMIT ? `) result, err := createStmt.Exec("244.157.42.179", 100) if err != nil { log.Fatal(err) } affected, _ := result.RowsAffected() fmt.Printf("Created table with %d rows\n", affected) } ``` -------------------------------- ### Create Minimal Configuration with NewNoOpsConfig Source: https://context7.com/uber/athenadriver/llms.txt Initializes a configuration without hardcoded credentials, relying on AWS SDK default resolution or environment variables. ```go package main import ( "database/sql" "fmt" "os" drv "github.com/uber/athenadriver/go" ) func main() { // Enable AWS SDK to load config from ~/.aws/config and ~/.aws/credentials os.Setenv("AWS_SDK_LOAD_CONFIG", "1") // Create minimal config and set required fields conf := drv.NewNoOpsConfig() conf.SetOutputBucket("s3://my-query-results-bucket/") conf.SetRegion("us-east-2") // Optionally set a specific AWS profile conf.SetAWSProfile("my-profile") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() var value int _ = db.QueryRow("SELECT 456").Scan(&value) fmt.Println("Value:", value) // Output: Value: 456 } ``` -------------------------------- ### Custom Pretty Print with Style and Render Options Source: https://context7.com/uber/athenadriver/llms.txt Allows custom formatting of query results using specified styles and render formats. The `style` parameter accepts predefined styles like 'StyleColoredGreenWhiteOnBlack', and `renderFormat` accepts 'csv', 'html', 'markdown', or 'table'. A `width` parameter controls the output width. ```go drv.PrettyPrintSQLColsRows(rows5, "StyleColoredGreenWhiteOnBlack", "table", 100) ``` -------------------------------- ### Show table properties and creation DDL Source: https://github.com/uber/athenadriver/blob/master/resources/UndocumentedAthena.md Use SHOW statements to inspect table properties or retrieve the original CREATE TABLE statement. ```sql SHOW TBLPROPERTIES sampledb.elb_logs ``` ```sql SHOW CREATE TABLE `sampledb.elb_logs` ``` ```sql createtab_stmt CREATE EXTERNAL TABLE `sampledb.elb_logs`( `request_timestamp` string COMMENT '', `elb_name` string COMMENT '', ... `ssl_protocol` string COMMENT '') ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( 'input.regex'='([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*):([0-9]*) ([.0-9]*) ([.0-9]*) ([.0-9]*) (-|[0-9]*) (-|[0-9]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) ([^ ]*) (- |[^ ]*)\" ("[^"]*") ([A-Z0-9-]+) ([A-Za-z0-9.-]*)$') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://athena-examples-us-east-2/elb/plaintext' TBLPROPERTIES ( 'transient_lastDdlTime'='1480278335') ``` -------------------------------- ### Execute SELECT Queries with Query and QueryRow Source: https://context7.com/uber/athenadriver/llms.txt Demonstrates standard database/sql methods for retrieving single or multiple rows from Athena. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Single row query with QueryRow var url string err := db.QueryRow("SELECT url FROM sampledb.elb_logs LIMIT 1").Scan(&url) if err != nil { log.Fatal(err) } fmt.Println("URL:", url) // Multiple rows query with Query rows, err := db.Query("SELECT request_timestamp, elb_name, url FROM sampledb.elb_logs LIMIT 5") if err != nil { log.Fatal(err) } defer rows.Close() for rows.Next() { var timestamp, elbName, url string if err := rows.Scan(×tamp, &elbName, &url); err != nil { log.Fatal(err) } fmt.Printf("Time: %s, ELB: %s, URL: %s\n", timestamp, elbName, url) } if err := rows.Err(); err != nil { log.Fatal(err) } } ``` -------------------------------- ### Query with Table Output and Styling Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Queries table schema information, displays the query cost, and formats the output as a table using the 'default' style. The results are stored in a specified S3 bucket. ```bash $ athenareader -b s3://henrywutest/ -q 'desc sampledb.elb_logs' -m -o table -y default query cost: 0.0 USD, scanned data: 0 B, qid: 96daa3d2-db82-4aa6-8c86-0bf05fd772dc +-------------------------+-----------+---------+ | COL_NAME | DATA_TYPE | COMMENT | +-------------------------+-----------+---------+ | request_timestamp | string | | | elb_name | string | | | request_ip | string | | | request_port | int | | | backend_ip | string | | | backend_port | int | | | request_processing_time | double | | | backend_processing_time | double | | | client_response_time | double | | | elb_response_code | string | | | backend_response_code | string | | | received_bytes | bigint | | | sent_bytes | bigint | | | request_verb | string | | | url | string | | | protocol | string | | | user_agent | string | | | ssl_cipher | string | | | ssl_protocol | string | | +-------------------------+-----------+---------+ ``` -------------------------------- ### Run Tests and Linters Source: https://github.com/uber/athenadriver/blob/master/resources/CONTRIBUTING.md Ensure that all tests and linters pass before submitting changes. Note that `make lint` may not function if not using the specified Go minor version. ```bash make test make lint ``` -------------------------------- ### Run Unit Tests with Coverage Source: https://github.com/uber/athenadriver/blob/master/README.md Execute unit tests and generate a coverage report. Filter out tests with 100% coverage to identify areas needing improvement. ```bash $ go test -coverprofile=coverage.out github.com/uber/athenadriver/go && \ go tool cover -func=coverage.out |grep -v 100.0% ``` -------------------------------- ### Run Integration Test and Simulate Network Failure Source: https://github.com/uber/athenadriver/blob/master/README.md Execute the concurrency test, redirect output to a log file, and simulate a network outage to observe error handling. ```bash ./concurrency > examples/perf/concurrency.output.`date +"%Y-%m-%d-%H-%M-%S"`.log ``` -------------------------------- ### Execute Queries with Timeout and Cancellation in Go Source: https://context7.com/uber/athenadriver/llms.txt Use context.WithTimeout for query timeouts and context.WithCancel for manual query cancellation. This helps manage costs by stopping expensive queries. ```go package main import ( "context" "database/sql" "fmt" "log" "time" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Create context with 30-second timeout ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() // Query will be automatically cancelled if it exceeds timeout rows, err := db.QueryContext(ctx, "SELECT COUNT(*) FROM sampledb.elb_logs") if err != nil { if err == context.DeadlineExceeded { fmt.Println("Query timed out and was cancelled") return } log.Fatal(err) } defer rows.Close() var count int64 if rows.Next() { rows.Scan(&count) fmt.Printf("Total rows: %d\n", count) } // Manual cancellation example ctx2, cancel2 := context.WithCancel(context.Background()) go func() { time.Sleep(2 * time.Second) fmt.Println("Cancelling query...") cancel2() // Cancel after 2 seconds }() _, err = db.QueryContext(ctx2, "SELECT * FROM sampledb.large_table") if err == context.Canceled { fmt.Println("Query was cancelled successfully") } } ``` -------------------------------- ### Execute Parameterized Queries in Go Source: https://github.com/uber/athenadriver/blob/master/README.md Use '?' as placeholders in queries and escape string or byte slice arguments with drv.FormatString or drv.FormatBytes. ```go query := "SELECT request_timestamp, elb_name FROM sampledb.elb_logs WHERE url=? limit 1" args := []any{drv.FormatString("https://www.example.com/jobs/878")} rows, err := db.Query(query, args) if err != nil { return } println(drv.ColsRowsToCSV(rows)) ``` -------------------------------- ### Import athenadriver Package Source: https://github.com/uber/athenadriver/blob/master/README.md Import the athenadriver package into your Go project to utilize its functionalities. ```go import athenadriver "github.com/uber/athenadriver/go" ``` -------------------------------- ### Manage Execution Results with DB.ExecContext Source: https://github.com/uber/athenadriver/blob/master/README.md Demonstrates using ExecContext and Exec to perform DDL and DML operations, including retrieving rows affected. ```go package main import ( "context" "database/sql" drv "github.com/uber/athenadriver/go" ) func main() { // 1. Set AWS Credential in Driver Config. var conf *drv.Config var err error if conf, err = drv.NewDefaultConfig("s3://myqueryresults/", "us-east-2", "DummyAccessID", "DummySecretAccessKey"); err != nil { panic(err) } // 2. Open Connection. db, _ := sql.Open(drv.DriverName, conf.Stringify()) // 3. Execute and print results if _, err = db.ExecContext(context.Background(), "DROP TABLE IF EXISTS sampledb.urls"); err != nil { panic(err) } var result sql.Result if result, err = db.Exec("CREATE TABLE sampledb.urls AS "+ "SELECT url FROM sampledb.elb_logs where request_ip=? limit ?", "244.157.42.179", 1); err != nil { panic(err) } println(result.RowsAffected()) if result, err = db.Exec("INSERT INTO sampledb.urls VALUES (?),(?),(?)", "abc", "efg", "xyz"); err != nil { panic(err) } println(result.RowsAffected()) println(result.LastInsertId()) // not supported by Athena } ``` -------------------------------- ### athenareader Help Text Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Displays the available command-line options and their descriptions for the athenareader tool. ```text -a Enable admin mode, so database write(create/drop) is allowed at athenadriver level -b string Athena resultset output bucket (default "s3://qr-athena-query-result-prod/Henry/") -d string The database you want to query (default "default") -m Enable moneywise mode to display the query cost as the first line of the output -o string Output format(options: table, markdown, csv, html) (default "csv") -q string The SQL query string or a file containing SQL string (default "select 1") -r Display rows only, don't show the first row as columninfo -v Print the current version and exit -y string Output rendering style (default "default") ``` -------------------------------- ### Enable Tally Metrics with StatsD Source: https://github.com/uber/athenadriver/blob/master/README.md Configure a tally scope with a StatsD reporter and attach it to the context for metrics collection. ```go package main import ( "context" "database/sql" "io" "log" "time" "github.com/cactus/go-statsd-client/statsd" tallystatsd "github.com/uber-go/tally/v4/statsd" drv "github.com/uber/athenadriver/go" "github.com/uber-go/tally/v4" ) func newScope() (tally.Scope, io.Closer) { statter, _ := statsd.NewBufferedClient("127.0.0.1:8125", "stats", 100*time.Millisecond, 1440) reporter := tallystatsd.NewReporter(statter, tallystatsd.Options{ SampleRate: 1.0, }) scope, closer := tally.NewRootScope(tally.ScopeOptions{ Prefix: "my_test_metrics_service", Tags: map[string]string{}, Reporter: reporter, }, time.Second) return scope, closer } func main() { // 1. Set AWS Credential in Driver Config. conf, _ := drv.NewDefaultConfig("s3://query-results-bucket-test/", "us-east-2", "dummy-to-be-replaced", "dummy-to-be-replaced") // 2. Open Connection. dsn := conf.Stringify() db, _ := sql.Open(drv.DBDriverName, dsn) // 3. Query cancellation after 2 seconds // Create tally scope scope, _ := newScope() // Create context and attach tally scope with context ctx, _ := context.WithTimeout(context.Background(), 5*time.Second) ctx = context.WithValue(ctx, drv.MetricsKey, scope) rows, err := db.QueryContext(ctx, "select count(*) from sampledb.elb_logs") if err != nil { log.Fatal(err) return } defer rows.Close() } ``` -------------------------------- ### NewDefaultConfig - Create Driver Configuration Source: https://context7.com/uber/athenadriver/llms.txt Creates a new driver configuration with explicit AWS credentials, region, and S3 output bucket. ```APIDOC ## NewDefaultConfig - Create Driver Configuration Creates a new driver configuration with AWS credentials, region, and S3 output bucket for query results. This is the primary way to configure the Athena driver with explicit credentials. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { // Create configuration with S3 bucket, region, and AWS credentials conf, err := drv.NewDefaultConfig( "s3://my-query-results-bucket/", // S3 bucket for query results "us-east-2", // AWS region "AKIAIOSFODNN7EXAMPLE", // AWS Access Key ID "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", // Secret Access Key ) if err != nil { log.Fatal(err) } // Open database connection using the configuration db, err := sql.Open(drv.DriverName, conf.Stringify()) if err != nil { log.Fatal(err) } defer db.Close() // Execute a simple query var result int err = db.QueryRow("SELECT 123").Scan(&result) if err != nil { log.Fatal(err) } fmt.Println("Result:", result) // Output: Result: 123 } ``` ``` -------------------------------- ### Query All Advanced Data Types in Go Source: https://github.com/uber/athenadriver/blob/master/README.md Demonstrates selecting and printing various advanced data types supported by Athena using the athenadriver. Ensure AWS credentials and S3 bucket are configured. ```go package main import ( "context" "database/sql" drv "github.com/uber/athenadriver/go" ) func main() { // 1. Set AWS Credential in Driver Config. conf, err := drv.NewDefaultConfig("s3://myqueryresults/", "us-east-2", "DummyAccessID", "DummySecretAccessKey") if err != nil { panic(err) } // 2. Open Connection. dsn := conf.Stringify() db, _ := sql.Open(drv.DriverName, dsn) // 3. Query and print results query := "SELECT JSON '\"Hello Athena\"', " + "ST_POINT(-74.006801, 40.70522), " + "ROW(1, 2.0), INTERVAL '2' DAY, " + "INTERVAL '3' MONTH, " + "TIME '01:02:03.456', " + "TIME '01:02:03.456 America/Los_Angeles', " + "TIMESTAMP '2001-08-22 03:04:05.321 America/Los_Angeles';" rows, err := db.Query(query) if err != nil { panic(err) } defer rows.Close() println(drv.ColsRowsToCSV(rows)) } ``` -------------------------------- ### Enable moneywise mode for cost estimation Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md The -m flag displays the query cost in USD before the result set. ```bash $ athenareader -b s3://athena-query-result -q 'select count(*) as cnt from sampledb.elb_logs' -m query cost: 0.00184898369752772851 USD cnt 1356206 ``` -------------------------------- ### Configure missing value handling Source: https://context7.com/uber/athenadriver/llms.txt Define how the driver interprets NULL or missing values in query results. Options include returning empty strings, type-appropriate defaults, or nil values. ```go package main import ( "database/sql" "fmt" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") // Option 1: Return empty string for missing values (default) conf.SetMissingAsEmptyString(true) // Option 2: Return type-appropriate default values // (0 for integers, false for booleans, "" for strings, etc.) conf.SetMissingAsEmptyString(false) conf.SetMissingAsDefault(true) // Option 3: Return nil for missing values conf.SetMissingAsEmptyString(false) conf.SetMissingAsDefault(false) conf.SetMissingAsNil(true) // Option 4: Strict mode - raise error on missing values conf.SetMissingAsEmptyString(false) conf.SetMissingAsDefault(false) conf.SetMissingAsNil(false) db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Handle potentially NULL values with sql.NullString rows, _ := db.Query("SELECT optional_field FROM sampledb.data_with_nulls") defer rows.Close() for rows.Next() { var field sql.NullString rows.Scan(&field) if field.Valid { fmt.Println("Value:", field.String) } else { fmt.Println("Value is NULL") } } } ``` -------------------------------- ### Pretty Print Query Results as Fancy Table Source: https://context7.com/uber/athenadriver/llms.txt Displays query results in a colored table format directly to standard output. Ensure the `rows` object is valid. ```go drv.PrettyPrintFancy(rows2) // Colored table format ``` -------------------------------- ### Parameterized Queries with FormatString Source: https://context7.com/uber/athenadriver/llms.txt Uses native Athena parameterization. FormatString and FormatBytes are required to properly escape string arguments. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Query with parameterized values using FormatString for proper escaping query := "SELECT request_timestamp, elb_name FROM sampledb.elb_logs WHERE url = ? LIMIT 1" // FormatString escapes special characters and wraps in single quotes urlArg := drv.FormatString("https://www.example.com/jobs/878") rows, err := db.Query(query, urlArg) if err != nil { log.Fatal(err) } defer rows.Close() // Print results as CSV fmt.Println(drv.ColsRowsToCSV(rows)) // Output: // request_timestamp,elb_name // 2015-01-06T04:03:01.351843Z,elb_demo_006 // Example with timestamp typecast query2 := "SELECT * FROM sampledb.events WHERE created > ? LIMIT 10" timestampArg := fmt.Sprintf("TIMESTAMP %s", drv.FormatString("2024-07-01 00:00:00")) rows2, _ := db.Query(query2, timestampArg) defer rows2.Close() } ``` -------------------------------- ### Configure Athena Workgroups with Tags in Go Source: https://context7.com/uber/athenadriver/llms.txt Set up Athena workgroups with custom tags for cost tracking and resource management. The driver can automatically create workgroups if they do not exist. ```go package main import ( "database/sql" "fmt" "log" drv "github.com/uber/athenadriver/go" ) func main() { conf, _ := drv.NewDefaultConfig("s3://my-bucket/", "us-east-2", "AccessID", "SecretKey") // Create tags for cost tracking tags := drv.NewWGTags() tags.AddTag("Team", "data-engineering") tags.AddTag("Project", "analytics") tags.AddTag("Environment", "production") // Create workgroup with tags wg := drv.NewWG("analytics_workgroup", nil, tags) // Set workgroup in config if err := conf.SetWorkGroup(wg); err != nil { log.Fatal(err) } // Allow automatic workgroup creation if it doesn't exist conf.SetWGRemoteCreationAllowed(true) db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() // Queries will now run under the specified workgroup rows, _ := db.Query("SELECT url FROM sampledb.elb_logs LIMIT 3") defer rows.Close() for rows.Next() { var url string rows.Scan(&url) fmt.Println(url) } // To disable automatic workgroup creation (workgroup must exist) conf.SetWGRemoteCreationAllowed(false) } ``` -------------------------------- ### NewNoOpsConfig - Create Minimal Configuration Source: https://context7.com/uber/athenadriver/llms.txt Creates a minimal configuration without credentials, useful when using AWS CLI configuration or SDK default credential resolution. ```APIDOC ## NewNoOpsConfig - Create Minimal Configuration Creates a minimal configuration without credentials, useful when using AWS CLI configuration or SDK default credential resolution. You must set the output bucket and region separately. ```go package main import ( "database/sql" "fmt" "os" drv "github.com/uber/athenadriver/go" ) func main() { // Enable AWS SDK to load config from ~/.aws/config and ~/.aws/credentials os.Setenv("AWS_SDK_LOAD_CONFIG", "1") // Create minimal config and set required fields conf := drv.NewNoOpsConfig() conf.SetOutputBucket("s3://my-query-results-bucket/") conf.SetRegion("us-east-2") // Optionally set a specific AWS profile conf.SetAWSProfile("my-profile") db, _ := sql.Open(drv.DriverName, conf.Stringify()) defer db.Close() var value int _ = db.QueryRow("SELECT 456").Scan(&value) fmt.Println("Value:", value) // Output: Value: 456 } ``` ``` -------------------------------- ### Listen for Metrics via Netcat Source: https://github.com/uber/athenadriver/blob/master/README.md Use netcat to listen for UDP metrics packets sent by the StatsD reporter. ```bash nc 8125 -l -u ``` -------------------------------- ### Query with HTML Output Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Executes a query to describe a table and formats the output as an HTML table. This is suitable for embedding query results in web pages. ```bash $ athenareader -b s3://henrywutest/ -q 'desc sampledb.elb_logs' -o html
| col_name | data_type | comment | ``` -------------------------------- ### Simple Query with Default CSV Output Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Executes a simple SQL query against a specified database and outputs the results in CSV format. This is the default output format. ```bash $ athenareader -d sampledb -q "select request_timestamp,elb_name from elb_logs limit 2" request_timestamp,elb_name 2015-01-03T00:00:00.516940Z,elb_demo_004 2015-01-03T00:00:00.902953Z,elb_demo_004 ``` -------------------------------- ### Pretty Print Query Results as CSV Source: https://context7.com/uber/athenadriver/llms.txt Formats query results as a CSV output. Ensure the `rows` object is valid. ```go drv.PrettyPrintCSV(rows4) // CSV format ``` -------------------------------- ### Query with Table Output and Red Style Source: https://github.com/uber/athenadriver/blob/master/athenareader/README.md Queries table schema information and formats the output as a table using the 'StyleColoredRedWhiteOnBlack' style. The results are stored in a specified S3 bucket. ```bash $ athenareader -b s3://henrywutest/ -q 'desc sampledb.elb_logs' -m -y StyleColoredRedWhiteOnBlack -o table ```
|---|