### Example: Publish Release 18.0.0 RC1 Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md An example command to publish version 18.0.0 RC1 using the release script and a GitHub token. ```bash GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release.sh 18.0.0 1 ``` -------------------------------- ### Example: Verify Release 18.0.0 RC1 Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md An example command to verify release 18.0.0 RC1 using the verify_rc.sh script. ```bash dev/release/verify_rc.sh 18.0.0 1 ``` -------------------------------- ### Install Go Assembly Tools Source: https://github.com/apache/arrow-go/blob/main/parquet/internal/utils/_lib/README.md Installs the necessary tools for generating Go assembly from C assembly. Ensure these tools are available before proceeding with assembly generation. ```bash go get -tool github.com/klauspost/asmfmt/cmd/asmfmt go get -tool github.com/minio/asm2plan9s go get -tool github.com/minio/c2goasm ``` -------------------------------- ### Generate Protocol Buffer Go Files Source: https://github.com/apache/arrow-go/blob/main/arrow/util/messages/README.md Navigate to the arrow/util directory and execute the protoc command to generate Go files from the types.proto definition. This command requires the protobuf compiler (protoc) to be installed. ```bash cd go/arrow/util/ protoc -I ./ --go_out=./messages ./messages/types.proto ``` -------------------------------- ### Conventional Commit Examples Source: https://github.com/apache/arrow-go/blob/main/CONTRIBUTING.md Examples of commit messages following the Conventional Commits specification, including scope and type. ```text fix(arrow/cdata): handle empty structs in C data interface ``` ```text ci: update CI environment ``` ```text feat(parquet): support new encoding type ``` -------------------------------- ### Verify Release Candidate Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md Use the verify_rc.sh script to check the integrity and correctness of a release candidate. This script requires curl, gpg, shasum (or sha256sum/sha512sum), and tar. It can automatically install Go if not present. ```bash dev/release/verify_rc.sh ${VERSION} ${RC} ``` -------------------------------- ### Release RC with GitHub Token Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md Execute the release candidate script with your GitHub token and the desired RC number. This is an example for releasing RC1. ```bash GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release_rc.sh 1 ``` -------------------------------- ### Write JSON to Output File Source: https://github.com/apache/arrow-go/blob/main/parquet/cmd/parquet_reader/README.md Redirect the JSON output to a file using the --output flag. The example shows writing to 'data.json' and then using 'jq' to pretty-print the content. ```bash $ ./parquet_reader --no-metadata --json --output=data.json v0.7.1.parquet $ jq . data.json [ { "carat": 0.23, "cut": "Ideal", "color": "E", "clarity": "SI2", "depth": 61.5, "table": 55, "price": 326, "x": 3.95, ... ``` -------------------------------- ### Write and Read Parquet File in Go Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates writing a simple Parquet file with two columns (int64 and byte array) and then reading it back using the parquet/file package. Includes schema definition, batch writing, and batch reading. ```go import ( "fmt" "log" "os" "github.com/apache/arrow-go/v18/parquet" "github.com/apache/arrow-go/v18/parquet/compress" "github.com/apache/arrow-go/v18/parquet/file" "github.com/apache/arrow-go/v18/parquet/schema" ) // Build a schema with two primitive columns node, _ := schema.NewPrimitiveNode("id", parquet.Repetitions.Required, parquet.Types.Int64, -1, -1) node2, _ := schema.NewPrimitiveNode("name", parquet.Repetitions.Optional, parquet.Types.ByteArray, -1, -1) root, _ := schema.NewGroupNode("schema", parquet.Repetitions.Required, schema.FieldList{node, node2}, -1) sc := schema.NewSchema(root) // Write f, _ := os.CreateTemp("", "example-*.parquet") fname := f.Name() fw := file.NewParquetWriter(f, sc.Root(), file.WithWriterProps(parquet.NewWriterProperties(parquet.WithCompression(compress.Codecs.Zstd))), ) rgw := fw.AppendRowGroup() idWriter, _ := rgw.NextColumn() idWriter.(*file.Int64ColumnChunkWriter).WriteBatch([]int64{10, 20, 30}, nil, nil) rgw.Close() fw.Close() f.Close() // Read pqFile, err := file.OpenParquetFile(fname, false) if err != nil { log.Fatal(err) } defer pqFile.Close() defer os.Remove(fname) fmt.Printf("NumRowGroups=%d NumCols=%d NumRows=%d\n", pqFile.NumRowGroups(), pqFile.MetaData().Schema.NumColumns(), pqFile.MetaData().NumRows, ) rg := pqFile.RowGroup(0) colRdr, _ := rg.Column(0) int64Rdr := colRdr.(*file.Int64ColumnChunkReader) vals := make([]int64, 3) n, _, _ := int64Rdr.ReadBatch(3, vals, nil, nil) fmt.Printf("Read %d values: %v\n", n, vals[:n]) ``` -------------------------------- ### Build and Read Multi-Column RecordBatch in Go Source: https://context7.com/apache/arrow-go/llms.txt Shows how to create a record builder for a schema with multiple columns (Int32 and Float64) and populate it. `NewRecordBatch()` finalizes and resets the builder. ```go import ( "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "fmt" ) pool := memory.NewGoAllocator() schema := arrow.NewSchema([]arrow.Field{ {Name: "f1-i32", Type: arrow.PrimitiveTypes.Int32}, {Name: "f2-f64", Type: arrow.PrimitiveTypes.Float64}, }, nil) b := array.NewRecordBuilder(pool, schema) deferr b.Release() b.Field(0).(*array.Int32Builder).AppendValues([]int32{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) b.Field(1).(*array.Float64Builder).AppendValues([]float64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) rec := b.NewRecordBatch() deferr rec.Release() for i, col := range rec.Columns() { fmt.Printf("column[%d] %q: %v\n", i, rec.ColumnName(i), col) } // Output: // column[0] "f1-i32": [1 2 3 4 5 6 7 8 9 10] // column[1] "f2-f64": [1 2 3 4 5 6 7 8 9 10] ``` -------------------------------- ### Clone Repository and Prepare Release Candidate Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md Clone the Apache Arrow Go repository and run the release candidate script. This script prepares the necessary artifacts for a release candidate. Ensure you are not running this on your fork. ```bash git clone git@github.com:apache/arrow-go.git dev/release/release_rc.sh ${RC} ``` -------------------------------- ### Open FlightSQL Connection with database/sql Source: https://github.com/apache/arrow-go/blob/main/arrow/flight/flightsql/driver/README.md Demonstrates how to open a connection to a FlightSQL backend using the standard Go database/sql package. Ensure the driver is imported and the DSN is correctly formatted. The connection should be deferred closed. ```go import ( "database/sql" "time" _ "github.com/apache/arrow-go/v18/arrow/flight/flightsql" ) // Open the connection to an SQLite backend db, err := sql.Open("flightsql", "flightsql://localhost:12345?timeout=5s") if err != nil { panic(err) } // Make sure we close the connection to the database defer db.Close() // Use the connection e.g. for querying rows, err := db.Query("SELECT * FROM mytable") if err != nil { panic(err) } // ... ``` -------------------------------- ### Publish Release to Apache.org Source: https://github.com/apache/arrow-go/blob/main/dev/release/README.md Run the release script with your GitHub token, the version, and the RC number to publish the release artifacts to apache.org. ```bash GH_TOKEN=${YOUR_GITHUB_TOKEN} dev/release/release.sh ${VERSION} ${RC} ``` -------------------------------- ### Parquet Reader Usage Source: https://github.com/apache/arrow-go/blob/main/parquet/cmd/parquet_reader/README.md Displays the available command-line options for the parquet_reader tool, including flags for help, metadata control, output formatting, and column selection. ```bash ./parquet_reader -h Parquet Reader (version 0.1.20220629.1846) Usage: parquet_reader -h | --help parquet_reader [--only-metadata] [--no-metadata] [--no-memory-map] [--json] [--csv] [--output=FILE] [--print-key-value-metadata] [--int96-timestamp] [--columns=COLUMNS] Options: -h --help Show this screen. --print-key-value-metadata Print out the key-value metadata. [default: false] --only-metadata Stop after printing metadata, no values. --no-metadata Do not print metadata. --output=FILE Specify output file for data. [default: -] --no-memory-map Disable memory mapping the file. --int96-timestamp Parse INT96 as TIMESTAMP for legacy support. --json Format output as JSON instead of text. --csv Format output as CSV instead of text. --columns=COLUMNS Specify a subset of columns to print, comma delimited indexes. ``` -------------------------------- ### Configure Flight SQL Driver with DriverConfig Source: https://github.com/apache/arrow-go/blob/main/arrow/flight/flightsql/driver/README.md Use DriverConfig to set connection parameters like address, token, timeout, and custom parameters, then generate the DSN. ```golang package main import ( "database/sql" "log" "time" "github.com/apache/arrow-go/v18/arrow/flight/flightsql" ) func main() { config := flightsql.DriverConfig{ Address: "localhost:12345", Token: "your token", Timeout: 10 * time.Second, Params: map[string]string{ "my-custom-parameter": "foobar", }, } db, err := sql.Open("flightsql", config.DSN()) if err != nil { log.Fatalf("open failed: %v", err) } defer db.Close() ... } ``` -------------------------------- ### Write and Read Parquet Files with Arrow Go Source: https://context7.com/apache/arrow-go/llms.txt Shows how to write Arrow RecordBatches to a Parquet file and then read them back. Ensure correct schema definition and use appropriate writer/reader properties for compression and schema storage. ```go import ( "bytes" "context" "fmt" "log" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "github.com/apache/arrow-go/v18/parquet" "github.com/apache/arrow-go/v18/parquet/compress" "github.com/apache/arrow-go/v18/parquet/file" "github.com/apache/arrow-go/v18/parquet/pqarrow" ) schema := arrow.NewSchema([]arrow.Field{ {Name: "id", Type: arrow.PrimitiveTypes.Int32, Nullable: false}, {Name: "name", Type: arrow.BinaryTypes.String, Nullable: false}, {Name: "scores", Type: arrow.ListOf(arrow.PrimitiveTypes.Float32), Nullable: false}, }, nil) // --- Write --- buf := &bytes.Buffer{} writerProps := parquet.NewWriterProperties(parquet.WithCompression(compress.Codecs.Snappy)) arrowWriterProps := pqarrow.NewArrowWriterProperties(pqarrow.WithStoreSchema()) w, err := pqarrow.NewFileWriter(schema, buf, writerProps, arrowWriterProps) if err != nil { log.Fatal(err) } rb := array.NewRecordBuilder(memory.DefaultAllocator, schema) rb.Field(0).(*array.Int32Builder).AppendValues([]int32{1, 2, 3}, nil) rb.Field(1).(*array.StringBuilder).AppendValues([]string{"Alice", "Bob", "Carol"}, nil) lb := rb.Field(2).(*array.ListBuilder) vb := lb.ValueBuilder().(*array.Float32Builder) for i := 0; i < 3; i++ { lb.Append(true); vb.AppendValues([]float32{1.0, 2.0}, nil) } rec := rb.NewRecordBatch() rb.Release() if err := w.Write(rec); err != nil { log.Fatal(err) } rec.Release() if err := w.Close(); err != nil { log.Fatal(err) } fmt.Printf("Parquet bytes: %d\n", buf.Len()) // --- Read --- pqReader, err := file.NewParquetReader(bytes.NewReader(buf.Bytes())) if err != nil { log.Fatal(err) } defer pqReader.Close() arReader, err := pqarrow.NewFileReader(pqReader, pqarrow.ArrowReadProperties{BatchSize: 2}, memory.DefaultAllocator) if err != nil { log.Fatal(err) } recReader, err := arReader.GetRecordReader(context.TODO(), nil, nil) // nil = all cols, all row groups if err != nil { log.Fatal(err) } defer recReader.Release() for recReader.Next() { r := recReader.RecordBatch() idCol := r.Column(0).(*array.Int32) nameCol := r.Column(1).(*array.String) for i := 0; i < int(r.NumRows()); i++ { fmt.Printf("%d %s\n", idCol.Value(i), nameCol.Value(i)) } } // Output: // Parquet bytes: // 1 Alice // 2 Bob // 3 Carol ``` -------------------------------- ### Enable TLS with DriverConfig Source: https://github.com/apache/arrow-go/blob/main/arrow/flight/flightsql/driver/README.md Configure TLS settings directly within DriverConfig, including enabling TLS, specifying a custom configuration name, and providing the TLS configuration. ```golang ... config := flightsql.DriverConfig{ Address: "localhost:12345", TLSEnabled: true, TLSConfigName: "myconfig", TLSConfig: &tls.Config{ MinVersion: tls.VersionTLS12, }, } dsn := config.DSN() ... ``` -------------------------------- ### Handling ARM Instructions with WORD Syntax Source: https://github.com/apache/arrow-go/blob/main/parquet/internal/utils/_lib/README.md After c2goasm, some ARM instructions need to be explicitly defined using WORD syntax. Unrecognized instructions will be commented out and require manual addition. ```asm // stp x29, x30, [sp, #-48] WORD $0x11007c48 // add w8, w2, #31 ``` -------------------------------- ### Create and Stream Arrow Tables from Records Source: https://context7.com/apache/arrow-go/llms.txt Use `NewTableFromRecords` to combine record batches into a table and `NewTableReader` to re-chunk tables for streaming consumption. Ensure to release resources using `defer`. ```go import ( "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "fmt" ) pool := memory.NewGoAllocator() schema := arrow.NewSchema([]arrow.Field{ {Name: "intField", Type: arrow.PrimitiveTypes.Int64, Nullable: false}, {Name: "stringField", Type: arrow.BinaryTypes.String, Nullable: false}, {Name: "floatField", Type: arrow.PrimitiveTypes.Float64, Nullable: true}, }, nil) b := array.NewRecordBuilder(memory.DefaultAllocator, schema) defer b.Release() b.Field(0).(*array.Int64Builder).AppendValues([]int64{1, 2, 3, 4, 5}, nil) b.Field(1).(*array.StringBuilder).AppendValues([]string{"a", "b", "c", "d", "e"}, nil) b.Field(2).(*array.Float64Builder).AppendValues([]float64{1, 0, 3, 0, 5}, []bool{true, false, true, false, true}) rec := b.NewRecordBatch() defer rec.Release() tbl := array.NewTableFromRecords(schema, []arrow.RecordBatch{rec}) defer tbl.Release() fmt.Printf("Rows=%d Cols=%d\n", tbl.NumRows(), tbl.NumCols()) // Stream with batch size 2 tr := array.NewTableReader(tbl, 2) defer tr.Release() n := 0 for tr.Next() { r := tr.RecordBatch() fmt.Printf("batch[%d] intField: %v\n", n, r.Column(0)) n++ } // Output: // Rows=5 Cols=3 // batch[0] intField: [1 2] // batch[1] intField: [3 4] // batch[2] intField: [5] ``` -------------------------------- ### Build Large Int64 Array with Explicit Allocator Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates building a large array using a specific memory allocator. Ensure to call Release() on the builder and the array when done to manage memory. ```go import ( "github.com/apache/arrow-go/v18/arrow/memory" "github.com/apache/arrow-go/v18/arrow/array" "fmt" ) pool := memory.NewGoAllocator() // Build 10M element array with explicit allocator builder := array.NewInt64Builder(pool) defer builder.Release() for i := int64(0); i < 10_000_000; i++ { builder.Append(i) } arr := builder.NewInt64Array() defer arr.Release() fmt.Printf("len=%d, first=%d, last=%d\n", arr.Len(), arr.Value(0), arr.Value(arr.Len()-1)) // Output: len=10000000, first=0, last=9999999 ``` -------------------------------- ### Build Static Memory Library Source: https://github.com/apache/arrow-go/blob/main/arrow/memory/_lib/CMakeLists.txt Creates a static library named 'memory' from the 'memory.c' source file. ```cmake add_library(memory STATIC memory.c) ``` -------------------------------- ### Copy Record Batches using arrio.Copy in Go Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates using arrio.Copy to pipe Arrow RecordBatches between a CSV/Flight/IPC source and an IPC sink. This function enables zero-code-change composition of various I/O formats. ```go import ( "bytes" "log" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/arrio" "github.com/apache/arrow-go/v18/arrow/ipc" "github.com/apache/arrow-go/v18/arrow/memory" ) pool := memory.NewGoAllocator() schema := arrow.NewSchema([]arrow.Field{ {Name: "v", Type: arrow.PrimitiveTypes.Int32}, }, nil) b := array.NewRecordBuilder(pool, schema) defer b.Release() b.Field(0).(*array.Int32Builder).AppendValues([]int32{10, 20, 30}, nil) rec := b.NewRecordBatch() defer rec.Release() // Pipe CSV/Flight/IPC source → IPC sink via arrio.Copy var out bytes.Buffer ipcWriter := ipc.NewWriter(&out, ipc.WithSchema(schema)) rdr, _ := array.NewRecordReader(schema, []arrow.RecordBatch{rec}) defer rdr.Release() // arrio.Copy accepts any arrio.Reader and arrio.Writer n, err := arrio.Copy(ipcWriter, rdr) if err != nil { log.Fatal(err) } if err := ipcWriter.Close(); err != nil { log.Fatal(err) } _ = n // number of records copied ``` -------------------------------- ### Implement Arrow Flight gRPC Server in Go Source: https://context7.com/apache/arrow-go/llms.txt Provides a base for implementing Arrow Flight gRPC services. Embed `flight.BaseFlightServer` and override methods as needed. Can be used with `flight.NewFlightServer` or registered with an existing gRPC server. ```go import ( "context" "fmt" "log" "net" "github.com/apache/arrow-go/v18/arrow/flight" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" ) // Server srv := flight.NewFlightServer() srv.Init("localhost:0") // port 0 → OS assigns a free port type myService struct{ flight.BaseFlightServer } srv.RegisterFlightService(&myService{}) go srv.Serve() defer srv.Shutdown() // Client conn, err := grpc.NewClient(srv.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { log.Fatal(err) } defer conn.Close() client := flight.NewFlightServiceClient(conn) _, err = client.GetFlightInfo(context.Background(), &flight.FlightDescriptor{ Type: flight.DescriptorPATH, Path: []string{"my-dataset"}, }) // BaseFlightServer returns Unimplemented for methods not overridden fmt.Println("err:", err) // err: rpc error: code = Unimplemented desc = method GetFlightInfo not implemented // Register against an existing grpc.Server s := grpc.NewServer() listener, _ := net.Listen("tcp", "localhost:0") svc := struct{ flight.BaseFlightServer }{} flight.RegisterFlightServiceServer(s, &svc) go s.Serve(listener) defer s.Stop() ``` -------------------------------- ### Define Arrow Schema with Various Field Types Source: https://context7.com/apache/arrow-go/llms.txt Illustrates creating an Arrow schema with different field types including primitive, binary, list, and struct types. Remember to handle nullability as specified. ```go import ( "github.com/apache/arrow-go/v18/arrow" "fmt" ) schema := arrow.NewSchema([]arrow.Field{ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: false}, {Name: "name", Type: arrow.BinaryTypes.String, Nullable: false}, {Name: "score", Type: arrow.PrimitiveTypes.Float64, Nullable: true}, {Name: "tags", Type: arrow.ListOf(arrow.BinaryTypes.String), Nullable: true}, {Name: "meta", Type: arrow.StructOf( arrow.Field{Name: "region", Type: arrow.BinaryTypes.String}, arrow.Field{Name: "tier", Type: arrow.PrimitiveTypes.Int32}, ), Nullable: true}, }, nil) fmt.Println("Fields:", schema.NumFields()) for i, f := range schema.Fields() { fmt.Printf(" [%d] %s: %s (nullable=%v)\n", i, f.Name, f.Type, f.Nullable) } // Output: // Fields: 5 // [0] id: int64 (nullable=false) // [1] name: utf8 (nullable=false) // [2] score: float64 (nullable=true) // [3] tags: list (nullable=true) // [4] meta: struct (nullable=true) ``` -------------------------------- ### Build and Slice Int64 and List Arrays in Go Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates building typed Arrow arrays (Int64, List) with nullability and slicing existing arrays. Builders are reusable after calling `New*Array()`. ```go import ( "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "fmt" ) pool := memory.NewGoAllocator() // Int64 with a null b := array.NewInt64Builder(pool) deferr b.Release() b.AppendValues([]int64{1, 2, 3, -1, 5}, []bool{true, true, true, false, true}) ints := b.NewInt64Array() deferr ints.Release() fmt.Printf("array = %v\n", ints) // [1 2 3 (null) 5] fmt.Printf("NullN = %d\n", ints.NullN()) // 1 // List lb := array.NewListBuilder(pool, arrow.PrimitiveTypes.Int64) deferr lb.Release() vb := lb.ValueBuilder().(*array.Int64Builder) vb.Reserve(7) lb.Append(true); vb.Append(0); vb.Append(1); vb.Append(2) // [0,1,2] lb.AppendNull() // null lb.Append(true); vb.Append(3); vb.Append(4) // [3,4] lists := lb.NewArray().(*array.List) deferr lists.Release() fmt.Printf("lists = %v\n", lists) // [[0 1 2] (null) [3 4]] // Slicing (zero-copy) sl := array.NewSlice(ints, 1, 4).(*array.Int64) deferr sl.Release() fmt.Printf("slice = %v\n", sl) // [2 3 (null)] ``` -------------------------------- ### Set C Standard and Add Static Libraries Source: https://github.com/apache/arrow-go/blob/main/arrow/math/_lib/CMakeLists.txt Configures the C standard to 99 and defines static libraries for float64, int64, and uint64 operations. This is used to build the core math components of the Arrow library. ```cmake set(CMAKE_C_STANDARD 99) add_library(memory STATIC float64.c int64.c uint64.c) ``` -------------------------------- ### Register and Use Custom TLS Configuration Manually Source: https://github.com/apache/arrow-go/blob/main/arrow/flight/flightsql/driver/README.md Manually register a custom TLS configuration with a specific name and then reference it in the DSN. This allows reusing the same TLS configuration across multiple DSNs. ```golang myconfig := &tls.Config{MinVersion: tls.VersionTLS12} if err := flightsql.RegisterTLSConfig("myconfig", myconfig); err != nil { ... } dsn := "flightsql://localhost:12345?tls=myconfig" ... ``` -------------------------------- ### Benchmark Summing with All Architecture Optimizations Disabled Source: https://github.com/apache/arrow-go/blob/main/README.md Run benchmarks for summing arrays with all architecture extensions disabled, forcing the use of a pure Go implementation. This is useful for baseline performance comparison. ```sh $ INTEL_DISABLE_EXT=ALL go test -bench=8192 -run=. ./math goos: darwin goarch: amd64 pkg: github.com/apache/arrow-go/arrow/math BenchmarkFloat64Funcs_Sum_8192-8 200000 10285 ns/op 6371.41 MB/s BenchmarkInt64Funcs_Sum_8192-8 500000 3892 ns/op 16837.37 MB/s BenchmarkUint64Funcs_Sum_8192-8 500000 3929 ns/op 16680.00 MB/s PASS ok github.com/apache/arrow-go/arrow/math 6.179s ``` -------------------------------- ### Iterate Over Multiple RecordBatches in Go Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates wrapping a slice of `RecordBatch` objects into an `array.RecordReader` for sequential iteration. This is the standard interface for Arrow readers and IPC writers. ```go import ( "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "fmt" "log" ) pool := memory.NewGoAllocator() schema := arrow.NewSchema([]arrow.Field{ {Name: "x", Type: arrow.PrimitiveTypes.Int32}, }, nil) b := array.NewRecordBuilder(pool, schema) deferr b.Release() b.Field(0).(*array.Int32Builder).AppendValues([]int32{1, 2, 3}, nil) rec1 := b.NewRecordBatch() deferr rec1.Release() b.Field(0).(*array.Int32Builder).AppendValues([]int32{4, 5, 6}, nil) rec2 := b.NewRecordBatch() deferr rec2.Release() rdr, err := array.NewRecordReader(schema, []arrow.RecordBatch{rec1, rec2}) if err != nil { log.Fatal(err) } deferr rdr.Release() n := 0 for rdr.Next() { rec := rdr.RecordBatch() fmt.Printf("batch[%d]: %v\n", n, rec.Column(0)) n++ } // Output: // batch[0]: [1 2 3] // batch[1]: [4 5 6] ``` -------------------------------- ### Write CSV with Apache Arrow Go Source: https://context7.com/apache/arrow-go/llms.txt Serializes Arrow record batches into CSV format. Requires a schema and an output writer. Options include specifying a header. ```go import ( "bytes" "fmt" "strings" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" arrowcsv "github.com/apache/arrow-go/v18/arrow/csv" ) // Writing CSV schema := arrow.NewSchema([]arrow.Field{ {Name: "id", Type: arrow.PrimitiveTypes.Int64}, {Name: "name", Type: arrow.BinaryTypes.String}, {Name: "score", Type: arrow.PrimitiveTypes.Float64}, }, nil) var buf bytes.Buffer w := arrowcsv.NewWriter(&buf, schema, arrowcsv.WithHeader(true)) // (build a record batch then write it) // w.Write(rec); w.Flush() ``` -------------------------------- ### Register and Invoke Custom Compute Function in Arrow Go Source: https://context7.com/apache/arrow-go/llms.txt Demonstrates registering a custom scalar function 'add_42' and invoking it against Arrow arrays. Ensure the compute context and registry are properly set up before calling CallFunction. ```go import ( "context" "fmt" "log" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/compute" "github.com/apache/arrow-go/v18/arrow/compute/exec" "github.com/apache/arrow-go/v18/arrow/memory" ) pool := memory.NewGoAllocator() execCtx := compute.DefaultExecCtx() ctx := compute.SetExecCtx(context.Background(), execCtx) // Register a custom scalar function: add 42 to every Int8 add42 := compute.NewScalarFunction("add_42", compute.Arity{NArgs: 1}, compute.FunctionDoc{Summary: "input + 42", ArgNames: []string{"input"}}) if err := add42.AddNewKernel( []exec.InputType{{Kind: exec.InputExact, Type: arrow.PrimitiveTypes.Int8}}, exec.NewOutputType(arrow.PrimitiveTypes.Int8), func(ctx *exec.KernelCtx, span *exec.ExecSpan, result *exec.ExecResult) error { for i, x := range span.Values[0].Array.Buffers[1].Buf { result.Buffers[1].Buf[i] = x + 42 } return nil }, nil, ); err != nil { log.Fatal(err) } execCtx.Registry.AddFunction(add42, true) // Build input ib := array.NewInt8Builder(pool) for i := 0; i < 8; i++ { ib.Append(int8(i)) } input := ib.NewArray() ib.Release() // Invoke out, err := compute.CallFunction(ctx, "add_42", nil, &compute.ArrayDatum{Value: input.Data()}) if err != nil { log.Fatal(err) } result := array.NewInt8Data(out.(*compute.ArrayDatum).Value) fmt.Println(result.Int8Values()) // Output: [42 43 44 45 46 47 48 49] ``` -------------------------------- ### Connect to FlightSQL using ADBC Driver Source: https://github.com/apache/arrow-go/blob/main/README.md Use this snippet to connect to a FlightSQL database using the Go database/sql interface and the ADBC FlightSQL driver. Ensure the ADBC driver is imported. ```golang import ( "database/sql" _ "github.com/apache/arrow-adbc/go/adbc/sqldriver/flightsql" ) func main() { dsn := "uri=grpc://localhost:12345;username=mickeymouse;password=p@55w0RD" db, err := sql.Open("flightsql", dsn) ... } ``` -------------------------------- ### Read Parquet to Text Output Source: https://github.com/apache/arrow-go/blob/main/parquet/cmd/parquet_reader/README.md Demonstrates reading a Parquet file and outputting its data in a plain text format, suppressing metadata display. ```bash $ ./parquet_reader --no-metadata v0.7.1.parquet carat |cut |color |clarity |depth |table |price |x |y |z |__index_level_0__ | 0.230000 |Ideal |E |SI2 |61.500000 |55.000000 |326 |3.950000 |3.980000 |2.430000 |0 | 0.210000 |Premium |E |SI1 |59.800000 |61.000000 |326 |3.890000 |3.840000 |2.310000 |1 | 0.230000 |Good |E |VS1 |56.900000 |65.000000 |327 |4.050000 |4.070000 |2.310000 |2 | 0.290000 |Premium |I |VS2 |62.400000 |58.000000 |334 |4.200000 |4.230000 |2.630000 |3 | 0.310000 |Good |J |SI2 |63.300000 |58.000000 |335 |4.340000 |4.350000 |2.750000 |4 | 0.240000 |Very Good |J |VVS2 |62.800000 |57.000000 |336 |3.940000 |3.960000 |2.480000 |5 | 0.240000 |Very Good |I |VVS1 |62.300000 |57.000000 |336 |3.950000 |3.980000 |2.470000 |6 | 0.260000 |Very Good |H |SI1 |61.900000 |55.000000 |337 |4.070000 |4.110000 |2.530000 |7 | 0.220000 |Fair |E |VS2 |65.100000 |61.000000 |337 |3.870000 |3.780000 |2.490000 |8 | 0.230000 |Very Good |H |VS1 |59.400000 |61.000000 |338 |4.000000 |4.050000 |2.390000 |9 | ``` -------------------------------- ### Configure CMake Build System Source: https://github.com/apache/arrow-go/blob/main/arrow/memory/_lib/CMakeLists.txt Sets the minimum required CMake version and defines the project name. It also specifies the C standard to be used for compilation. ```cmake cmake_minimum_required(VERSION 3.6) project(memory-func) set(CMAKE_C_STANDARD 99) ``` -------------------------------- ### Convert Go Structs to Arrow Records and Vice Versa Source: https://context7.com/apache/arrow-go/llms.txt Utilize `arreflect` for reflection-based conversion between Go slices/structs and Arrow arrays/record batches. `InferSchema` derives schemas at compile time. Ensure resources are released. ```go import ( "github.com/apache/arrow-go/v18/arrow/array/arreflect" "github.com/apache/arrow-go/v18/arrow/memory" "fmt" ) type Measurement struct { Sensor string `arrow:"sensor"` Value float64 `arrow:"value"` } // Infer schema at compile time schema, err := arreflect.InferSchema[Measurement]() if err != nil { panic(err) } fmt.Println("Schema:", schema) mem := memory.NewGoAllocator() // Convert Go structs → RecordBatch rows := []Measurement{{"temp-1", 23.5}, {"temp-2", 19.8}} rec, err := arreflect.RecordFromSlice(rows, mem) if err != nil { panic(err) } defer rec.Release() fmt.Println("Rows:", rec.NumRows()) fmt.Println("Col sensor:", rec.Column(0)) fmt.Println("Col value:", rec.Column(1)) // Convert RecordBatch → Go structs got, err := arreflect.RecordToSlice[Measurement](rec) if err != nil { panic(err) } for _, m := range got { fmt.Printf("%s: %.1f\n", m.Sensor, m.Value) } // Scalar slice with options arr, err := arreflect.FromSlice( []string{"red", "green", "red", "blue", "green"}, mem, arreflect.WithDict(), // encode as dictionary array ) if err != nil { panic(err) } defer arr.Release() fmt.Println("Dict type:", arr.DataType()) // dictionary // Output: // Schema: schema: // fields: 2 // - sensor: type=utf8 // - value: type=float64 // Rows: 2 // Col sensor: ["temp-1" "temp-2"] // Col value: [23.5 19.8] // temp-1: 23.5 // temp-2: 19.8 // Dict type: dictionary ``` -------------------------------- ### Read Parquet to CSV Source: https://github.com/apache/arrow-go/blob/main/parquet/cmd/parquet_reader/README.md Use the --csv flag to output the Parquet file content in CSV format to standard output. The --no-metadata flag suppresses metadata display. ```bash $ ./parquet_reader --no-metadata --csv v0.7.1.parquet "carat","cut","color","clarity","depth","table","price","x","y","z","__index_level_0__" 0.23,"Ideal","E","SI2",61.5,55,326,3.95,3.98,2.43,0 0.21,"Premium","E","SI1",59.8,61,326,3.89,3.84,2.31,1 0.23,"Good","E","VS1",56.9,65,327,4.05,4.07,2.31,2 0.29,"Premium","I","VS2",62.4,58,334,4.2,4.23,2.63,3 0.31,"Good","J","SI2",63.3,58,335,4.34,4.35,2.75,4 0.24,"Very Good","J","VVS2",62.8,57,336,3.94,3.96,2.48,5 0.24,"Very Good","I","VVS1",62.3,57,336,3.95,3.98,2.47,6 0.26,"Very Good","H","SI1",61.9,55,337,4.07,4.11,2.53,7 0.22,"Fair","E","VS2",65.1,61,337,3.87,3.78,2.49,8 0.23,"Very Good","H","VS1",59.4,61,338,4,4.05,2.39,9 ``` -------------------------------- ### ARM64 Constant Reference Conversion Source: https://github.com/apache/arrow-go/blob/main/parquet/internal/utils/_lib/README.md Replace 'adrp' and 'str/ldr/mov' sequences for constant references with '#define' macros and 'VMOVD' or 'VMOVQ' instructions. ```asm #define LCPI0_0 $0x0000000200000001 #define LCPI0_1 $0xffffffe2ffffffe1 ``` ```asm #define LCPI0_48L $0x0000000d00000008 #define LCPI0_48H $0x0000001700000012 ``` -------------------------------- ### Benchmark Summing with No Architecture Optimizations Disabled Source: https://github.com/apache/arrow-go/blob/main/README.md Run benchmarks for summing arrays with no architecture extensions disabled, effectively enabling AVX2 and SSE4 optimizations. This test is useful for measuring peak performance. ```sh $ INTEL_DISABLE_EXT=NONE go test -bench=8192 -run=. ./math goos: darwin goarch: amd64 pkg: github.com/apache/arrow-go/arrow/math BenchmarkFloat64Funcs_Sum_8192-8 2000000 687 ns/op 95375.41 MB/s BenchmarkInt64Funcs_Sum_8192-8 2000000 719 ns/op 91061.06 MB/s BenchmarkUint64Funcs_Sum_8192-8 2000000 691 ns/op 94797.29 MB/s PASS ok github.com/apache/arrow-go/arrow/math 6.444s ``` -------------------------------- ### Read Parquet to JSON Source: https://github.com/apache/arrow-go/blob/main/parquet/cmd/parquet_reader/README.md Use the --json flag to output the Parquet file content in JSON format to standard output. The --no-metadata flag suppresses metadata display. ```bash $ ./parquet_reader --no-metadata --json v0.7.1.parquet [{"carat":0.23,"cut":"Ideal","color":"E","clarity":"SI2","depth":61.5,"table":55,"price":326,"x":3.95,"y":3.98,"z":2.43,"__index_level_0__":0},{"carat":0.21,"cut":"Premium","color":"E","clarity":"SI1","depth":59.8,"table":61,"price":326,"x":3.89,"y":3.84,"z":2.31,"__index_level_0__":1},{"carat":0.23,"cut":"Good","color":"E","clarity":"VS1","depth":56.9,"table":65,"price":327,"x":4.05,"y":4.07,"z":2.31,"__index_level_0__":2},{"carat":0.29,"cut":"Premium","color":"I","clarity":"VS2","depth":62.4,"table":58,"price":334,"x":4.2,"y":4.23,"z":2.63,"__index_level_0__":3},{"carat":0.31,"cut":"Good","color":"J","clarity":"SI2","depth":63.3,"table":58,"price":335,"x":4.34,"y":4.35,"z":2.75,"__index_level_0__":4},{"carat":0.24,"cut":"Very Good","color":"J","clarity":"VVS2","depth":62.8,"table":57,"price":336,"x":3.94,"y":3.96,"z":2.48,"__index_level_0__":5},{"carat":0.24,"cut":"Very Good","color":"I","clarity":"VVS1","depth":62.3,"table":57,"price":336,"x":3.95,"y":3.98,"z":2.47,"__index_level_0__":6},{"carat":0.26,"cut":"Very Good","color":"H","clarity":"SI1","depth":61.9,"table":55,"price":337,"x":4.07,"y":4.11,"z":2.53,"__index_level_0__":7},{"carat":0.22,"cut":"Fair","color":"E","clarity":"VS2","depth":65.1,"table":61,"price":337,"x":3.87,"y":3.78,"z":2.49,"__index_level_0__":8},{"carat":0.23,"cut":"Very Good","color":"H","clarity":"VS1","depth":59.4,"table":61,"price":338,"x":4,"y":4.05,"z":2.39,"__index_level_0__":9}] ``` -------------------------------- ### ARM64 Vector Move Instruction Source: https://github.com/apache/arrow-go/blob/main/parquet/internal/utils/_lib/README.md Use 'VMOVD' for single 64-bit vector moves or 'VMOVQ' for double 128-bit vector moves, referencing defined constants. ```asm VMOVD LCPI0_0, v4 ``` ```asm VMOVQ LCPI0_48L, LCPI0_48H, V4 ``` -------------------------------- ### ARM64 Assembly Comment and Data Type Conversion Source: https://github.com/apache/arrow-go/blob/main/parquet/internal/utils/_lib/README.md Before running c2goasm, convert ARM64 comments from '//' to '#' and adjust data types like '.word' to '.long' and '.xword' to '.quad' to match x86-64 expectations. ```asm // stp x29, x30, [sp, #-48] WORD $0x11007c48 // add w8, w2, #31 ```