### Install Hamba Avro Source: https://github.com/hamba/avro/blob/main/README.md Use 'go get' to install the Hamba Avro library. Ensure your Go environment is updated if you encounter issues with the default branch name. ```shell go get github.com/hamba/avro/v2 ``` -------------------------------- ### Install Avro Struct Generator Source: https://github.com/hamba/avro/blob/main/README.md Install the avrogen command-line tool to generate Go structs from Avro schemas. Replace with the desired tag. ```shell go install github.com/hamba/avro/v2/cmd/avrogen@ ``` -------------------------------- ### Install Avro Schema Validator Source: https://github.com/hamba/avro/blob/main/README.md Install the avrosv command-line utility for Avro schema validation. Replace with the desired tag. ```shell go install github.com/hamba/avro/v2/cmd/avrosv@ ``` -------------------------------- ### Get Avrosv Help Source: https://github.com/hamba/avro/blob/main/README.md Display help information and available options for the avrosv command-line tool. ```shell avrosv -h ``` -------------------------------- ### Get Avrogen Help Source: https://github.com/hamba/avro/blob/main/README.md Display help information and available options for the avrogen command-line tool. ```shell avrogen -h ``` -------------------------------- ### Parse Avro Schema with Panic on Error Source: https://context7.com/hamba/avro/llms.txt Use `avro.MustParse` for package-level schema initialization where invalid schemas should cause a startup panic. This is a convenience wrapper around `avro.Parse`. ```go package main import "github.com/hamba/avro/v2" // Declare schema at package level — panics at startup if invalid. var userSchema = avro.MustParse(`{ "type": "record", "name": "User", "namespace": "com.example", "fields": [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"} ] }`) ``` -------------------------------- ### Marshal and Unmarshal Avro Data in Go Source: https://github.com/hamba/avro/blob/main/README.md Demonstrates how to define a Go struct for Avro, parse an Avro schema, marshal a Go struct into Avro data, and unmarshal Avro data back into a Go struct. Requires importing 'log' and 'fmt'. ```go type SimpleRecord struct { A int64 `avro:"a"` B string `avro:"b"` } schema, err := avro.Parse(`{ "type": "record", "name": "simple", "namespace": "org.hamba.avro", "fields" : [ {"name": "a", "type": "long"}, {"name": "b", "type": "string"} ] }`) if err != nil { log.Fatal(err) } in := SimpleRecord{A: 27, B: "foo"} data, err := avro.Marshal(schema, in) if err != nil { log.Fatal(err) } fmt.Println(data) // Outputs: [54 6 102 111 111] out := SimpleRecord{} err = avro.Unmarshal(schema, data, &out) if err != nil { log.Fatal(err) } fmt.Println(out) // Outputs: {27 foo} ``` -------------------------------- ### avrogen CLI Tool Source: https://context7.com/hamba/avro/llms.txt The `avrogen` command-line tool generates Go struct definitions from Avro schema files (`.avsc`). It supports various options for package naming, output files, and custom logical type mappings. ```APIDOC ## `avrogen` CLI Generates Go struct definitions from Avro schema files (`.avsc`), following the same type mapping rules as the codec. ### Installation ```shell go install github.com/hamba/avro/v2/cmd/avrogen@latest ``` ### Usage - **Basic**: `avrogen -pkg events -o ./events/types.go ./schemas/event.avsc` - **Dump to stdout**: `avrogen -pkg events ./schemas/event.avsc` - **Multiple struct tags**: `avrogen -pkg events -o ./events/types.go -tags json:snake,yaml:upper-camel ./schemas/event.avsc` - **Custom logical type mapping**: `avrogen -pkg events -o ./events/types.go -logical-type uuid,uuid.UUID,github.com/google/uuid ./schemas/event.avsc` - **Map date logical type to int32**: `avrogen -pkg events -o ./events/types.go -logical-type date,int32 ./schemas/event.avsc` - **Help**: `avrogen -h` ``` -------------------------------- ### Generate Go Structs from Avro Schemas with `avrogen` CLI Source: https://context7.com/hamba/avro/llms.txt Use the `avrogen` CLI to generate Go struct definitions from Avro schema files. This tool follows the same type mapping rules as the Avro codec and supports various options for package naming, output files, struct tags, and custom logical type mappings. ```shell # Install go install github.com/hamba/avro/v2/cmd/avrogen@latest # Basic usage: generate to a file avrogen -pkg events -o ./events/types.go ./schemas/event.avsc # Dump to stdout avrogen -pkg events ./schemas/event.avsc # Multiple struct tags avrogen -pkg events -o ./events/types.go -tags json:snake,yaml:upper-camel ./schemas/event.avsc # Custom logical type mapping (uuid → github.com/google/uuid.UUID) avrogen -pkg events -o ./events/types.go \ -logical-type uuid,uuid.UUID,github.com/google/uuid \ ./schemas/event.avsc # Map date logical type to int32 (built-in type, no import needed) avrogen -pkg events -o ./events/types.go \ -logical-type date,int32 \ ./schemas/event.avsc # All options avrogen -h ``` -------------------------------- ### Generate Go Structs from Avro Schema Source: https://github.com/hamba/avro/blob/main/README.md Use the avrogen tool to generate Go structs. Specify the package, output file, and optional tags. Omit -o FILE to output to stdout. ```shell avrogen -pkg avro -o bla.go -tags json:snake,yaml:upper-camel in.avsc ``` -------------------------------- ### Create Custom Avro API Instance with Freeze Source: https://context7.com/hamba/avro/llms.txt Use `avro.Config{}.Freeze()` to create a custom API instance with non-default settings like custom struct tag keys or union resolution behavior. The returned API has the same Marshal/Unmarshal methods as package-level functions. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) type Event struct { Type string `json:"type"` Payload string `json:"payload"` } func main() { api := avro.Config{ TagKey: "json", // use `json:"..."` tags instead of `avro:"..."` UnionResolutionError: true, // error on unresolvable union types MaxByteSliceSize: 4096, // restrict max string/bytes size to 4 KiB }.Freeze() schema := avro.MustParse(`{ "type": "record", "name": "Event", "fields": [ {"name": "type", "type": "string"}, {"name": "payload", "type": "string"} ] }`) event := Event{Type: "click", Payload: `{"x":10,"y":20}`} data, err := api.Marshal(schema, event) if err != nil { log.Fatal(err) } var out Event if err := api.Unmarshal(schema, data, &out); err != nil { log.Fatal(err) } fmt.Printf("%+v\n", out) // {Type:click Payload:{"x":10,"y":20}} } ``` -------------------------------- ### Check Schema Evolution Compatibility Source: https://context7.com/hamba/avro/llms.txt Use `avro.NewSchemaCompatibility` and its `Compatible` method to verify if a reader schema is compatible with a writer schema, adhering to Avro's schema evolution rules. ```go package main import ( "fmt" "github.com/hamba/avro/v2" ) func main() { writerSchema := avro.MustParse(`{ "type": "record", "name": "User", "fields": [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"} ] }`) // Reader adds a new optional field with a default — compatible. readerSchema := avro.MustParse(`{ "type": "record", "name": "User", "fields": [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "email", "type": ["null","string"], "default": null} ] }`) compat := avro.NewSchemaCompatibility() if err := compat.Compatible(readerSchema, writerSchema); err != nil { fmt.Println("incompatible:", err) } else { fmt.Println("schemas are compatible") // schemas are compatible } } ``` -------------------------------- ### Read and Write Avro Object Container Files (OCF) in Go Source: https://context7.com/hamba/avro/llms.txt Utilize `ocf.NewDecoder` and `ocf.NewEncoder` to handle Avro `.avro` container files. These files embed the schema and support block-level compression. ```go package main import ( "log" "os" "github.com/hamba/avro/v2" "github.com/hamba/avro/v2/ocf" ) type LogEntry struct { Timestamp int64 `avro:"ts"` Level string `avro:"level"` Message string `avro:"message"` } const logSchema = `{ "type": "record", "name": "LogEntry", "fields": [ {"name": "ts", "type": "long"}, {"name": "level", "type": "string"}, {"name": "message", "type": "string"} ] }` func writeAvroFile(path string, entries []LogEntry) error { f, err := os.Create(path) if err != nil { return err } defer f.Close() enc, err := ocf.NewEncoder(logSchema, f, ocf.WithCodec(ocf.ZStandard)) if err != nil { return err } for _, e := range entries { if err := enc.Encode(e); err != nil { return err } } return enc.Flush() } func readAvroFile(path string) ([]LogEntry, error) { f, err := os.Open(path) if err != nil { return nil, err } defer f.Close() dec, err := ocf.NewDecoder(f) if err != nil { return nil, err } defer dec.Close() var entries []LogEntry for dec.HasNext() { var e LogEntry if err := dec.Decode(&e); err != nil { return nil, err } entries = append(entries, e) } return entries, dec.Error() } func main() { entries := []LogEntry{ {Timestamp: 1700000000, Level: "INFO", Message: "service started"}, {Timestamp: 1700000001, Level: "WARN", Message: "high memory usage"}, } if err := writeAvroFile("/tmp/logs.avro", entries); err != nil { log.Fatal(err) } _ = avro.MustParse(logSchema) // ensure schema cached loaded, err := readAvroFile("/tmp/logs.avro") if err != nil { log.Fatal(err) } for _, e := range loaded { log.Printf("[%s] %s\n", e.Level, e.Message) } // [INFO] service started // [WARN] high memory usage } ``` -------------------------------- ### Map Built-in Go Type with Avrogen Source: https://github.com/hamba/avro/blob/main/README.md Map a logical type to a built-in Go type, omitting the import path. ```shell avrogen -pkg avro -o bla.go -logical-type date,int32 in.avsc ``` -------------------------------- ### Registry Client Interface Source: https://context7.com/hamba/avro/llms.txt The `registry.NewClient` function creates a client for interacting with the Confluent Schema Registry. The `Registry` interface provides methods to manage and retrieve schemas, and it integrates with `registry.NewDecoder` for decoding Avro messages. ```APIDOC ## `registry.NewClient` / `Registry` interface Provides a client for Confluent Schema Registry: look up schemas by ID or subject, create/check registration, and validate compatibility. Integrates with `registry.NewDecoder` for decoding Confluent wire-format payloads (magic byte + 4-byte schema ID prefix). ### Methods - **`NewClient(endpoint string)`** (func) - Creates a new Schema Registry client. - **`IsRegistered(ctx context.Context, subject string, schema string)`** (func) - Checks if a schema is registered for a given subject and returns its ID and schema object if found. - **`CreateSchema(ctx context.Context, subject string, schema string)`** (func) - Creates a new schema registration for a given subject and returns its ID and schema object. - **`IsCompatible(ctx context.Context, subject string, schema string)`** (func) - Checks if a new schema is compatible with the existing schema for a given subject. ### Usage Example ```go client, err := registry.NewClient("http://schema-registry:8081") // ... use client methods like IsRegistered, CreateSchema, IsCompatible dec := registry.NewDecoder(client) // ... use dec.Decode to decode Avro messages ``` ``` -------------------------------- ### Confluent Schema Registry Client for Go Source: https://context7.com/hamba/avro/llms.txt Use the `registry.NewClient` to interact with Confluent Schema Registry for schema registration, retrieval, and compatibility checks. It integrates with `registry.NewDecoder` for decoding Confluent wire-format messages. ```go package main import ( "context" "fmt" "log" "github.com/hamba/avro/v2" "github.com/hamba/avro/v2/registry" ) type Payment struct { ID string `avro:"id"` Amount float64 `avro:"amount"` } func main() { client, err := registry.NewClient("http://schema-registry:8081") if err != nil { log.Fatal(err) } ctx := context.Background() schemaJSON := `{ "type": "record", "name": "Payment", "fields": [ {"name": "id", "type": "string"}, {"name": "amount", "type": "double"} ] }` // Register (or retrieve) the schema. id, schema, err := client.IsRegistered(ctx, "payments-value", schemaJSON) if err != nil { id, schema, err = client.CreateSchema(ctx, "payments-value", schemaJSON) if err != nil { log.Fatal(err) } } fmt.Printf("schema id=%d type=%s\n", id, schema.Type()) // Decode a Confluent wire-format message from Kafka. dec := registry.NewDecoder(client) wireMsg := []byte{0x00, 0x00, 0x00, 0x00, byte(id) /* avro bytes */} var p Payment if err := dec.Decode(ctx, wireMsg, &p); err != nil { log.Fatal(err) } fmt.Printf("%+v\n", p) // Check schema evolution compatibility. newSchema := `{ "type": "record", "name": "Payment", "fields": [ {"name": "id", "type": "string"}, {"name": "amount", "type": "double"}, {"name": "currency", "type": ["null","string"], "default": null} ] }` ok, err := client.IsCompatible(ctx, "payments-value", newSchema) if err != nil { log.Fatal(err) } fmt.Println("compatible:", ok) // compatible: true _ = avro.DefaultConfig // avro package used for config access } ``` -------------------------------- ### avro.Config.Freeze Source: https://context7.com/hamba/avro/llms.txt Creates a custom frozen configuration (an `API`) with non-default settings such as a custom struct tag key, block length, union resolution behavior, or byte-slice size limits. The returned `API` has the same `Marshal`/`Unmarshal`/`NewEncoder`/`NewDecoder` methods as the package-level functions. ```APIDOC ## `avro.Config.Freeze` — Create a custom API instance ### Description Creates a custom frozen configuration (an `API`) with non-default settings such as a custom struct tag key, block length, union resolution behavior, or byte-slice size limits. The returned `API` has the same `Marshal`/`Unmarshal`/`NewEncoder`/`NewDecoder` methods as the package-level functions. ### Usage Example ```go api := avro.Config{ TagKey: "json", // use `json:"..."` tags instead of `avro:"..."` UnionResolutionError: true, // error on unresolvable union types MaxByteSliceSize: 4096, // restrict max string/bytes size to 4 KiB }.Freeze() // Use the custom API instance for marshaling and unmarshaling // data, err := api.Marshal(schema, event) // err := api.Unmarshal(schema, data, &out) ``` ``` -------------------------------- ### avro.MustParse Source: https://context7.com/hamba/avro/llms.txt A convenience wrapper around `avro.Parse` that panics if the schema parsing fails. Useful for initializing schemas at package level. ```APIDOC ## avro.MustParse — Parse a schema, panicking on error ### Description Convenience wrapper around `Parse` that panics instead of returning an error. Useful for package-level schema initialization. ### Function Signature ```go func MustParse(schema string) *Schema ``` ### Parameters #### Path Parameters - **schema** (string) - Required - The Avro schema in JSON format. ### Response #### Success Response (Schema) - **Schema** (*Schema) - The parsed Avro schema object. ### Behavior - Panics if the provided schema string is invalid. ``` -------------------------------- ### Construct Avro Record Schema Programmatically in Go Source: https://context7.com/hamba/avro/llms.txt Use the schema construction API to build Avro schemas in code instead of parsing JSON strings. This method is useful for dynamic schema generation. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) func main() { // Build: {"type":"record","name":"com.example.Product","fields":[ // {"name":"sku","type":"string"}, // {"name":"price","type":"double"}, // {"name":"tags","type":{"type":"array","items":"string"}} // ]} skuField, err := avro.NewField("sku", avro.NewPrimitiveSchema(avro.String, nil)) if err != nil { log.Fatal(err) } priceField, err := avro.NewField("price", avro.NewPrimitiveSchema(avro.Double, nil)) if err != nil { log.Fatal(err) } tagsField, err := avro.NewField("tags", avro.NewArraySchema(avro.NewPrimitiveSchema(avro.String, nil))) if err != nil { log.Fatal(err) } schema, err := avro.NewRecordSchema("Product", "com.example", []*avro.Field{skuField, priceField, tagsField}, avro.WithDoc("A product record"), ) if err != nil { log.Fatal(err) } fmt.Println(schema.FullName()) // com.example.Product fmt.Println(schema.String()) // canonical JSON form } ``` -------------------------------- ### Parse Avro Schema with Custom Cache and Namespace Source: https://context7.com/hamba/avro/llms.txt Utilize `avro.ParseWithCache` to parse schemas within a specific namespace and a provided `SchemaCache`. This allows for isolated schema management. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) func main() { cache := &avro.SchemaCache{} // Parse a shared address schema into the cache. _, err := avro.ParseWithCache(`{ "type": "record", "name": "Address", "namespace": "com.example", "fields": [ {"name": "street", "type": "string"}, {"name": "city", "type": "string"} ] }`, "com.example", cache) if err != nil { log.Fatal(err) } // Parse a schema that references Address. schema, err := avro.ParseWithCache(`{ "type": "record", "name": "Person", "namespace": "com.example", "fields": [ {"name": "name", "type": "string"}, {"name": "address", "type": "Address"} ] }`, "com.example", cache) if err != nil { log.Fatal(err) } fmt.Println(schema.Type()) // record } ``` -------------------------------- ### avro.Marshal Source: https://context7.com/hamba/avro/llms.txt Serializes a Go struct (or any compatible value) to an Avro binary byte slice using the provided schema. Struct fields are mapped via `avro:""` tags. ```APIDOC ## avro.Marshal — Encode a Go value to Avro bytes ### Description Serializes a Go struct (or any compatible value) to an Avro binary byte slice using the provided schema. Struct fields are mapped via `avro:""` tags. ### Usage ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) type Order struct { ID int64 `avro:"id"` Item string `avro:"item"` Quantity int32 `avro:"quantity"` Price float64 `avro:"price"` } var orderSchema = avro.MustParse(`{ "type": "record", "name": "Order", "fields": [ {"name": "id", "type": "long"}, {"name": "item", "type": "string"}, {"name": "quantity", "type": "int"}, {"name": "price", "type": "double"} ] }`) func main() { order := Order{ID: 1001, Item: "widget", Quantity: 5, Price: 9.99} data, err := avro.Marshal(orderSchema, order) if err != nil { log.Fatal(err) } fmt.Printf("encoded bytes: %v\n", data) } ``` ### Parameters - `schema` (*avro.Schema): The Avro schema to use for encoding. - `value` (any): The Go value to encode. ### Returns - `[]byte`: The Avro-encoded byte slice. - `error`: An error if the encoding fails. ``` -------------------------------- ### Parse Avro Schemas from Files Source: https://context7.com/hamba/avro/llms.txt Use `avro.ParseFiles` to read and parse multiple schema files, supporting inter-schema references. It returns the last schema parsed. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) func main() { // base-types.avsc defines reusable named types; event.avsc references them. schema, err := avro.ParseFiles("schemas/base-types.avsc", "schemas/event.avsc") if err != nil { log.Fatal(err) } fmt.Println(schema.Type()) // record } ``` -------------------------------- ### Compute Schema Fingerprints Source: https://context7.com/hamba/avro/llms.txt Utilize the `FingerprintUsing` method on an Avro schema to compute its fingerprint using supported algorithms like `MD5` or `CRC64Avro`. This is useful for schema identity comparison and registry lookups. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) func main() { schema := avro.MustParse(`{ "type": "record", "name": "Event", "fields": [{"name": "id", "type": "long"}] }`) md5fp, err := schema.FingerprintUsing(avro.MD5) if err != nil { log.Fatal(err) } crc64fp, err := schema.FingerprintUsing(avro.CRC64Avro) if err != nil { log.Fatal(err) } fmt.Printf("MD5: %x\n", md5fp) fmt.Printf("CRC64: %x\n", crc64fp) } ``` -------------------------------- ### Validate Schemas with References Source: https://github.com/hamba/avro/blob/main/README.md Validate multiple Avro schemas, including those that reference other schemas, by providing them in order. ```shell avrosv base-schema.avsc schema-withref.avsc ``` -------------------------------- ### avro.ParseFiles Source: https://context7.com/hamba/avro/llms.txt Reads and parses multiple Avro schema files, supporting inter-schema references. Returns the last schema parsed. ```APIDOC ## avro.ParseFiles — Parse schemas from files (supports inter-schema references) ### Description Reads and parses schema files in order, so later schemas can reference types defined in earlier files. Returns the last schema parsed. ### Function Signature ```go func ParseFiles(filenames ...string) (*Schema, error) ``` ### Parameters #### Path Parameters - **filenames** ([]string) - Required - A variadic list of file paths to Avro schema files. ### Response #### Success Response (Schema) - **Schema** (*Schema) - The last parsed Avro schema object. - **error** (error) - nil if parsing is successful. #### Error Response - **error** (error) - An error object if parsing fails. ``` -------------------------------- ### avrosv CLI Tool Source: https://context7.com/hamba/avro/llms.txt The `avrosv` command-line tool validates Avro schema files (`.avsc`). It's useful for CI/CD pipelines as it exits with code 0 for valid schemas and non-zero for invalid ones. ```APIDOC ## `avrosv` CLI Validates Avro schema files (`.avsc`) using the library's parser. Exit code `0` = valid; non-zero = invalid. Useful in CI/CD pipelines. ### Installation ```shell go install github.com/hamba/avro/v2/cmd/avrosv@latest ``` ### Usage - **Validate single schema**: `avrosv ./schemas/event.avsc` - **Validate multiple schemas**: `avrosv ./schemas/base-types.avsc ./schemas/event.avsc` - **Show parsed schema JSON**: `avrosv -dump ./schemas/event.avsc` - **Help**: `avrosv -h` ### Example (Invalid Schema) ```shell avrosv ./schemas/bad-schema.avsc; echo "exit: $?" # Output: # Error: avro: invalid default for field someString. not a string # exit: 2 ``` ``` -------------------------------- ### Marshal Go value to Avro bytes Source: https://context7.com/hamba/avro/llms.txt Serializes a Go struct to Avro binary data. Ensure struct fields are tagged with `avro:""` to map them to the schema. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) type Order struct { ID int64 `avro:"id" Item string `avro:"item" Quantity int32 `avro:"quantity" Price float64 `avro:"price" } var orderSchema = avro.MustParse(`{ "type": "record", "name": "Order", "fields": [ {"name": "id", "type": "long"}, {"name": "item", "type": "string"}, {"name": "quantity", "type": "int"}, {"name": "price", "type": "double"} ] }`) func main() { order := Order{ID: 1001, Item: "widget", Quantity: 5, Price: 9.99} data, err := avro.Marshal(orderSchema, order) if err != nil { log.Fatal(err) } fmt.Printf("encoded bytes: %v\n", data) // encoded bytes: [210 15 12 119 105 100 103 101 116 10 ...] } ``` -------------------------------- ### avro.Unmarshal Source: https://context7.com/hamba/avro/llms.txt Parses Avro binary data and stores the result into the pointed-to Go value. ```APIDOC ## avro.Unmarshal — Decode Avro bytes into a Go value ### Description Parses Avro binary data and stores the result into the pointed-to Go value. ### Usage ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) type Order struct { ID int64 `avro:"id"` Item string `avro:"item"` Quantity int32 `avro:"quantity"` Price float64 `avro:"price"` } var orderSchema = avro.MustParse(`{ "type": "record", "name": "Order", "fields": [ {"name": "id", "type": "long"}, {"name": "item", "type": "string"}, {"name": "quantity", "type": "int"}, {"name": "price", "type": "double"} ] }`) func main() { // Avro-encoded bytes (previously produced by Marshal) data := []byte{0xd2, 0x0f, 0x0c, 0x77, 0x69, 0x64, 0x67, 0x65, 0x74, 0x0a, 0x9a, 0x99, 0x13, 0xc0, 0xf5, 0x28, 0x23, 0x40} var order Order if err := avro.Unmarshal(orderSchema, data, &order); err != nil { log.Fatal(err) } fmt.Printf("%+v\n", order) } ``` ### Parameters - `schema` (*avro.Schema): The Avro schema to use for decoding. - `data` ([]byte): The Avro-encoded byte slice. - `ptr` (any): A pointer to the Go value to decode into. ### Returns - `error`: An error if the decoding fails. ``` -------------------------------- ### Allow Non-Spec-Compliant Schema Names with `avro.SkipNameValidation` Source: https://context7.com/hamba/avro/llms.txt Set the global `avro.SkipNameValidation` flag to `true` to disable strict Avro name validation. This allows parsing schemas that may not fully comply with the Avro naming specification, often produced by other Avro implementations. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) func main() { // The Java Avro library sometimes produces names that violate the spec. avro.SkipNameValidation = true defer func() { avro.SkipNameValidation = false }() schema, err := avro.Parse(`{ "type": "record", "name": "my-record", "fields": [{"name": "value", "type": "string"}] }`) if err != nil { log.Fatal(err) } fmt.Println(schema.Type()) // record } ``` -------------------------------- ### Implement Avro UnionConverter Interface Source: https://github.com/hamba/avro/blob/main/README.md Implement the UnionConverter interface for type-safe handling of Avro unions. Ensure pointer receivers for interface methods. ```go type UnionConverter interface { // FromAny payload decode into any of the mentioned types in the Union. FromAny(payload any) error // ToAny from the Union struct ToAny() (any, error) } ``` ```go const Schema = `{"name": "Payload", "type": "record", "fields": [{"name": "union", "type": ["int", {"type": "record", "name": "test", "fields" : [{"name": "a", "type": "long"}, {"name": "b", "type": "string"}]}]}]}` ``` ```go type Payload struct { Union *UnionRecord `avro:"union"` } ``` ```go type UnionRecord struct { Int *int Test *TestRecord } ``` ```go func (u *UnionRecord) ToAny() (any, error) { if u.Int != nil { return u.Int, nil } else if u.Test != nil { return u.Test, nil } return nil, errors.New("no value to encode") } ``` ```go func (u *UnionRecord) FromAny(payload any) error { switch t := payload.(type) { case int: u.Int = &t case TestRecord: u.Test = &t default: return errors.New("unknown type during decode of union") } return nil } ``` ```go type TestRecord struct { A int64 `avro:"a"` B string `avro:"b"` } ``` -------------------------------- ### Stream-encode multiple records with Encoder Source: https://context7.com/hamba/avro/llms.txt Creates an encoder to write successive Avro-encoded values to an `io.Writer`. This is efficient for encoding many records sequentially. ```go package main import ( "bytes" "fmt" "log" "github.com/hamba/avro/v2" ) type Metric struct { Name string `avro:"name" Value float64 `avro:"value" } func main() { schema := `{ "type": "record", "name": "Metric", "fields": [ {"name": "name", "type": "string"}, {"name": "value", "type": "double"} ] }` buf := &bytes.Buffer{} enc, err := avro.NewEncoder(schema, buf) if err != nil { log.Fatal(err) } metrics := []Metric{ {Name: "cpu_usage", Value: 72.5}, {Name: "mem_usage", Value: 58.3}, } for _, m := range metrics { if err := enc.Encode(m); err != nil { log.Fatal(err) } } fmt.Printf("encoded %d bytes\n", buf.Len()) } ``` -------------------------------- ### Register Custom Type Converters for Avro Source: https://context7.com/hamba/avro/llms.txt Use `avro.RegisterTypeConverters` with `avro.TypeConversionFuncs` to handle encoding/decoding of custom Go types. This is necessary for types that don't naturally map to Avro, especially when using `map[string]any` or `[]any` targets. ```go package main import ( "fmt" "log" "net" "github.com/hamba/avro/v2" ) func main() { // Encode net.IP ([]byte) as Avro string; decode Avro string back to net.IP. avro.RegisterTypeConverters(avro.TypeConversionFuncs{ AvroType: avro.String, EncoderTypeConversion: func(in any, _ avro.Schema) (any, error) { ip, ok := in.(net.IP) if !ok { return nil, fmt.Errorf("expected net.IP, got %T", in) } return ip.String(), nil }, DecoderTypeConversion: func(in any, _ avro.Schema) (any, error) { s, ok := in.(string) if !ok { return nil, fmt.Errorf("expected string, got %T", in) } return net.ParseIP(s), nil }, }) schema := avro.MustParse(`{ "type": "record", "name": "Host", "fields": [{"name": "ip", "type": "string"}] }`) type Host struct { IP any `avro:"ip"` } h := Host{IP: net.ParseIP("192.168.1.1")} data, err := avro.Marshal(schema, h) if err != nil { log.Fatal(err) } var out Host if err := avro.Unmarshal(schema, data, &out); err != nil { log.Fatal(err) } fmt.Printf("%T %v\n", out.IP, out.IP) // net.IP 192.168.1.1 } ``` -------------------------------- ### Stream-decode multiple records with Decoder Source: https://context7.com/hamba/avro/llms.txt Creates a decoder to read successive Avro-encoded values from an `io.Reader`. The loop terminates when `io.EOF` is encountered. ```go package main import ( "bytes" "errors" "fmt" "io" "log" "github.com/hamba/avro/v2" ) type Metric struct { Name string `avro:"name" Value float64 `avro:"value" } func main() { schema := `{ "type": "record", "name": "Metric", "fields": [ {"name": "name", "type": "string"}, {"name": "value", "type": "double"} ] }` // Simulate an incoming data stream buf := bytes.NewReader([]byte{ 0x12, 0x63, 0x70, 0x75, 0x5f, 0x75, 0x73, 0x61, 0x67, 0x65, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x52, 0x40, }) dec, err := avro.NewDecoder(schema, buf) if err != nil { log.Fatal(err) } for { var m Metric if err := dec.Decode(&m); errors.Is(err, io.EOF) { break } else if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", m) // {Name:cpu_usage Value:72.5} } } ``` -------------------------------- ### avro.ParseWithCache Source: https://context7.com/hamba/avro/llms.txt Parses an Avro schema using a provided custom cache and namespace. This allows for isolated schema caches. ```APIDOC ## avro.ParseWithCache — Parse a schema using a custom cache and namespace ### Description Parses a schema within a given namespace and a caller-managed `SchemaCache`, enabling isolated schema caches per application or avoiding global cache pollution. ### Function Signature ```go func ParseWithCache(schema string, namespace string, cache *SchemaCache) (*Schema, error) ``` ### Parameters #### Path Parameters - **schema** (string) - Required - The Avro schema in JSON format. - **namespace** (string) - Required - The namespace to associate with the schema. - **cache** (*SchemaCache) - Required - The custom schema cache to use. ### Response #### Success Response (Schema) - **Schema** (*Schema) - The parsed Avro schema object. - **error** (error) - nil if parsing is successful. #### Error Response - **error** (error) - An error object if parsing fails. ``` -------------------------------- ### Map Custom Logical Type with Avrogen Source: https://github.com/hamba/avro/blob/main/README.md Register custom logical type mappings for code generation. Format: avroLogicalType,goType[,importPath]. ```shell avrogen -pkg avro -o bla.go -logical-type uuid,uuid.UUID,github.com/google/uuid in.avsc ``` -------------------------------- ### avro.NewEncoder / (*Encoder).Encode Source: https://context7.com/hamba/avro/llms.txt Creates an encoder that writes successive Avro-encoded values to an `io.Writer`. Useful for encoding many records in sequence. ```APIDOC ## avro.NewEncoder / (*Encoder).Encode — Stream-encode multiple records ### Description Creates an encoder that writes successive Avro-encoded values to an `io.Writer`. Useful for encoding many records in sequence (e.g., to a network connection or file). ### Usage ```go package main import ( "bytes" "fmt" "log" "github.com/hamba/avro/v2" ) type Metric struct { Name string `avro:"name"` Value float64 `avro:"value"` } func main() { schema := `{ "type": "record", "name": "Metric", "fields": [ {"name": "name", "type": "string"}, {"name": "value", "type": "double"} ] }` buf := &bytes.Buffer{} enc, err := avro.NewEncoder(schema, buf) if err != nil { log.Fatal(err) } metrics := []Metric{ {Name: "cpu_usage", Value: 72.5}, {Name: "mem_usage", Value: 58.3}, } for _, m := range metrics { if err := enc.Encode(m); err != nil { log.Fatal(err) } } fmt.Printf("encoded %d bytes\n", buf.Len()) } ``` ### Methods - `avro.NewEncoder(schema string, w io.Writer) (*Encoder, error)`: Creates a new Avro encoder. - `(*Encoder).Encode(value any) error`: Encodes a single value to the writer. ``` -------------------------------- ### avro.NewDecoder / (*Decoder).Decode Source: https://context7.com/hamba/avro/llms.txt Creates a decoder that reads successive Avro-encoded values from an `io.Reader`. Returns `io.EOF` when the stream is exhausted. ```APIDOC ## avro.NewDecoder / (*Decoder).Decode — Stream-decode multiple records ### Description Creates a decoder that reads successive Avro-encoded values from an `io.Reader`. Returns `io.EOF` when the stream is exhausted. ### Usage ```go package main import ( "bytes" "errors" "fmt" "io" "log" "github.com/hamba/avro/v2" ) type Metric struct { Name string `avro:"name"` Value float64 `avro:"value"` } func main() { schema := `{ "type": "record", "name": "Metric", "fields": [ {"name": "name", "type": "string"}, {"name": "value", "type": "double"} ] }` // Simulate an incoming data stream buf := bytes.NewReader([]byte{ 0x12, 0x63, 0x70, 0x75, 0x5f, 0x75, 0x73, 0x61, 0x67, 0x65, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x52, 0x40, }) dec, err := avro.NewDecoder(schema, buf) if err != nil { log.Fatal(err) } for { var m Metric if err := dec.Decode(&m); errors.Is(err, io.EOF) { break } else if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", m) } } ``` ### Methods - `avro.NewDecoder(schema string, r io.Reader) (*Decoder, error)`: Creates a new Avro decoder. - `(*Decoder).Decode(ptr any) error`: Decodes the next value from the reader into the provided pointer. ``` -------------------------------- ### Register Named Types for Union Decoding into `any` Source: https://context7.com/hamba/avro/llms.txt Use `avro.Register` to associate Go types with Avro schema names, enabling decoding of unions into `any` (interface{}). Primitive types are pre-registered; named types like enums and records require explicit registration. ```go package main import ( "fmt" "log" "github.com/hamba/avro/v2" ) type ClickEvent struct { X int32 `avro:"x"` Y int32 `avro:"y"` } type ViewEvent struct { Page string `avro:"page"` } func main() { avro.Register("com.example.ClickEvent", ClickEvent{}) avro.Register("com.example.ViewEvent", ViewEvent{}) schema := avro.MustParse(`[ "null", {"type":"record","name":"ClickEvent","namespace":"com.example","fields":[ {"name":"x","type":"int"}, {"name":"y","type":"int"} ]}, {"type":"record","name":"ViewEvent","namespace":"com.example","fields":[ {"name":"page","type":"string"} ]} ]`) data := []byte{0x04, 0x14, 0x28} // ClickEvent{X:10, Y:20} var result any if err := avro.Unmarshal(schema, data, &result); err != nil { log.Fatal(err) } fmt.Printf("%T %+v\n", result, result) // avro_test.ClickEvent {X:10 Y:20} } ``` -------------------------------- ### Enable Name Validation Skipping in Go Source: https://github.com/hamba/avro/blob/main/README.md Globally disable Avro name validation in the Go library to parse schemas with invalid names. Unset after use. ```go avro.SkipNameValidation = true ``` -------------------------------- ### Validate Avro Schema Files with `avrosv` CLI Source: https://context7.com/hamba/avro/llms.txt The `avrosv` CLI validates `.avsc` schema files. It returns an exit code of 0 for valid schemas and non-zero for invalid ones, making it suitable for CI/CD pipelines. It can validate single schemas, multiple schemas with inter-references, and optionally dump the parsed schema JSON. ```shell # Install go install github.com/hamba/avro/v2/cmd/avrosv@latest # Validate a single schema avrosv ./schemas/event.avsc # Validate schemas with inter-references (parsed in order) avrosv ./schemas/base-types.avsc ./schemas/event.avsc # Invalid schema: prints error and exits non-zero avrosv ./schemas/bad-schema.avsc; echo "exit: $?" # Error: avro: invalid default for field someString. not a string # exit: 2 # Show parsed schema JSON avrosv -dump ./schemas/event.avsc # All options avrosv -h ```