### Install AvroTurf Gem Source: https://github.com/dasch/avro_turf/blob/master/README.md Add the AvroTurf gem to your application's Gemfile and install it using Bundler, or install it directly using the gem command. ```ruby gem 'avro_turf' ``` ```bash $ bundle ``` ```bash $ gem install avro_turf ``` -------------------------------- ### Initialize AvroTurf Messaging API Source: https://github.com/dasch/avro_turf/blob/master/README.md Instantiate the Messaging API by providing the URL of your Schema Registry. This setup is required before encoding or decoding messages. ```ruby require 'avro_turf/messaging' # You need to pass the URL of your Schema Registry. avro = AvroTurf::Messaging.new(registry_url: "http://my-registry:8081/") ``` -------------------------------- ### SchemaStore Usage with Nested Schemas (Pre-v1.0.0 Behavior) Source: https://github.com/dasch/avro_turf/blob/master/README.md Demonstrates how to load schemas into a SchemaStore and access them. Prior to AvroTurf v1.0.0, nested schemas defined within other schemas were directly resolvable. This example shows the older behavior which may no longer work. ```ruby store = AvroTurf::SchemaStore.new(path: 'my/schemas') store.load_schemas! # Accessing 'person' is correct and works fine. person = store.find('person', 'contacts') # my/schemas/contacts/person.avsc exists # Trying to access 'address' raises AvroTurf::SchemaNotFoundError address = store.find('address', 'contacts') # my/schemas/contacts/address.avsc is not found ``` -------------------------------- ### Configure WebMock for Test Schema Registry Source: https://context7.com/dasch/avro_turf/llms.txt Configures WebMock to stub any requests to the test schema registry URL and resets the registry between examples. ```ruby RSpec.configure do |config| config.before(:each) do stub_request(:any, /^http:\/\/registry\.test/).to_rack(TestSchemaRegistry) TestSchemaRegistry.clear # reset between examples end end ``` -------------------------------- ### Schema Definition with Nested Schema Reference Source: https://github.com/dasch/avro_turf/blob/master/README.md Example of an Avro schema ('person.avsc') that references another schema ('address') defined in a separate file. This allows for modular and reusable schema definitions. ```json // person.avsc { "name": "person", "type": "record", "fields": [ { "name": "full_name", "type": "string" }, { "name": "address", "type": "address" } ] } // address.avsc { "name": "address", "type": "record", "fields": [ { "name": "street", "type": "string" }, { "name": "city", "type": "string" } ] } ``` -------------------------------- ### Schema Definition Referencing an Array of Schemas Source: https://github.com/dasch/avro_turf/blob/master/README.md Example of an Avro schema ('person_list.avsc') that defines an array where each item is of the 'person' type, referencing the 'person' schema defined elsewhere. ```json // person_list.avsc { "type": "array", "items": "person" } ``` -------------------------------- ### Schema Definition with Nested Record Source: https://github.com/dasch/avro_turf/blob/master/README.md Example of a 'person' Avro schema containing a nested 'address' record. Note that in AvroTurf v1.0.0 and later, nested schemas like 'address' must be defined in their own .avsc files to be resolvable by `SchemaStore#find`. ```json { "name": "person", "namespace": "contacts", "type": "record", "fields": [ { "name": "address", "type": { "name": "address", "type": "record", "fields": [ { "name": "addr1", "type": "string" }, { "name": "addr2", "type": "string" }, { "name": "city", "type": "string" }, { "name": "zip", "type": "string" } ] } } ] } ``` -------------------------------- ### Initialize AvroTurf Encoder/Decoder Source: https://context7.com/dasch/avro_turf/llms.txt Create the main AvroTurf interface. Specify the path to schema files, an optional default namespace, and an optional compression codec. ```ruby require "avro_turf" # Schema files layout: # app/schemas/contacts/person.avsc # app/schemas/contacts/address.avsc avro = AvroTurf.new( schemas_path: "app/schemas/", namespace: "contacts", # optional default namespace codec: "deflate" # optional compression ) ``` -------------------------------- ### Initialize AvroTurf Source: https://github.com/dasch/avro_turf/blob/master/README.md Initialize AvroTurf by specifying the directory where your Avro schemas are stored. Schemas will be looked up from this directory. ```ruby # Schemas will be looked up from the specified directory. avro = AvroTurf.new(schemas_path: "app/schemas/") ``` -------------------------------- ### Initialize AvroTurf Messaging for Producer/Consumer Source: https://context7.com/dasch/avro_turf/llms.txt Initializes AvroTurf::Messaging with a schema registry URL and a local schemas path. This is used for encoding and decoding messages in streaming applications. ```ruby require "avro_turf/messaging" AvroTurf::Messaging.new( registry_url: "http://registry.test", schemas_path: "spec/fixtures/schemas/" ) ``` -------------------------------- ### Set up Authorized Fake Confluent Schema Registry Server for Testing Source: https://github.com/dasch/avro_turf/blob/master/README.md Inherit from FakeConfluentSchemaRegistryServer and configure host authorization for testing. This is necessary due to recent Sinatra updates and requires Sinatra to be added to your Gemfile or gemspec. Use RSpec and WebMock for stubbing. ```ruby require 'avro_turf/test/fake_confluent_schema_registry_server' require 'webmock/rspec' class AuthorizedFakeConfluentSchemaRegistryServer < FakeConfluentSchemaRegistryServer set :host_authorization, permitted_hosts: ['registry.example.com'] end # within an example let(:registry_url) { "http://registry.example.com" } before do stub_request(:any, /^#{registry_url}/).to_rack(AuthorizedFakeConfluentSchemaRegistryServer) AuthorizedFakeConfluentSchemaRegistryServer.clear end # Messaging objects created with the same registry_url will now use the fake server. ``` -------------------------------- ### Eagerly Load All Schemas with AvroTurf Source: https://context7.com/dasch/avro_turf/llms.txt Pre-loads and caches all schemas from the specified path. Call this at application boot to catch schema parse errors early. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") avro.load_schemas! # loads all *.avsc files recursively, raises on parse errors ``` -------------------------------- ### AvroTurf#load_schemas! Source: https://context7.com/dasch/avro_turf/llms.txt Eagerly loads and caches all schemas found under the specified `schemas_path`. It's recommended to call this at application boot to catch schema parsing errors early and avoid lazy-loading latency. ```APIDOC ## AvroTurf#load_schemas! — Eagerly load all schemas Pre-loads and caches every `.avsc` file found under `schemas_path`. Call this at application boot to surface schema parse errors early and avoid lazy-load latency during the first request. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") avro.load_schemas! # loads all *.avsc files recursively, raises on parse errors ``` ``` -------------------------------- ### Define a Test Schema Registry Source: https://context7.com/dasch/avro_turf/llms.txt Sets up a fake schema registry for testing purposes, allowing custom host authorization. ```ruby require "avro_turf/test/fake_confluent_schema_registry_server" require "webmock/rspec" class TestSchemaRegistry < FakeConfluentSchemaRegistryServer set :host_authorization, permitted_hosts: ["registry.test"] end ``` -------------------------------- ### Fetch Schema by Subject and Version Source: https://context7.com/dasch/avro_turf/llms.txt Fetches a schema from the registry by its subject name. Optionally specify a version; defaults to the latest if omitted. Returns the schema and its ID. ```ruby require "avro_turf/messaging" avro = AvroTurf::Messaging.new(registry_url: "http://schema-registry:8081/") schema, id = avro.fetch_schema(subject: "analytics.event") puts "Latest schema id: #{id}" schema_v1, id_v1 = avro.fetch_schema(subject: "analytics.event", version: 1) ``` -------------------------------- ### AvroTurf.new Source: https://context7.com/dasch/avro_turf/llms.txt Creates the primary interface for file-format Avro encoding and decoding. Schemas are resolved from .avsc files under schemas_path. Supports an optional codec for compressed output and an optional namespace applied to all schema lookups. ```APIDOC ## AvroTurf.new — Core encoder/decoder Creates the primary interface for file-format Avro encoding and decoding. Schemas are resolved from `.avsc` files under `schemas_path`. Supports an optional `codec` (`"deflate"`) for compressed output and an optional `namespace` applied to all schema lookups. ```ruby require "avro_turf" # Schema files layout: # app/schemas/contacts/person.avsc # app/schemas/contacts/address.avsc avro = AvroTurf.new( schemas_path: "app/schemas/", namespace: "contacts", # optional default namespace codec: "deflate" # optional compression ) ``` ``` -------------------------------- ### Register Schema by Name and Namespace Source: https://context7.com/dasch/avro_turf/llms.txt Pushes a local Avro schema (`.avsc` file) to the registry. The subject defaults to the schema's fully-qualified name but can be overridden. Returns the registered schema and its ID. ```ruby require "avro_turf/messaging" avro = AvroTurf::Messaging.new( registry_url: "http://schema-registry:8081/", schemas_path: "app/schemas/" ) # Register under the schema's own fullname schema, id = avro.register_schema(schema_name: "event", namespace: "analytics") # Register under a custom subject (e.g., Kafka topic value convention) schema, id = avro.register_schema( schema_name: "event", namespace: "analytics", subject: "my-kafka-topic-value" ) puts "Registered as id #{id}" ``` -------------------------------- ### Encode and Decode a Greeting Message Source: https://context7.com/dasch/avro_turf/llms.txt Demonstrates encoding a hash into binary Avro format using a specified schema name and then decoding the binary back into a hash. ```ruby binary = avro.encode({ "title" => "hello" }, schema_name: "greeting") decoded = avro.decode(binary) expect(decoded).to eq("title" => "hello") ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#fetch Source: https://context7.com/dasch/avro_turf/llms.txt Fetches a schema from the registry by its ID. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#fetch ### Description Retrieves the schema definition from the registry corresponding to the provided schema ID. ### Method `fetch(id)` ### Parameters * **id** (Integer) - The unique ID of the schema to fetch. ### Response * **schema_json** (String) - The JSON representation of the Avro schema. ### Request Example ```ruby registry.fetch(3) # => '{"type":"record","name":"greeting",...}' ``` ``` -------------------------------- ### AvroTurf::MutableSchemaStore - Runtime Schema Injection Source: https://context7.com/dasch/avro_turf/llms.txt Extends SchemaStore to allow adding schemas from in-memory hashes, useful for testing or dynamic schema handling. Schemas can be added before being used by AvroTurf. ```ruby store = AvroTurf::MutableSchemaStore.new(path: "app/schemas/") store.add_schema( "name" => "event", "namespace" => "analytics", "type" => "record", "fields" => [ { "name" => "event_type", "type" => "string" }, { "name" => "timestamp", "type" => "long" } ] ) avro = AvroTurf.new(schema_store: store) binary = avro.encode( { "event_type" => "page_view", "timestamp" => Time.now.to_i }, schema_name: "event", namespace: "analytics" ) ``` -------------------------------- ### AvroTurf::MutableSchemaStore Source: https://context7.com/dasch/avro_turf/llms.txt Extends `SchemaStore` to allow adding schemas directly from in-memory hashes, rather than only from disk files. This is particularly useful for testing or when schemas are received dynamically. ```APIDOC ## AvroTurf::MutableSchemaStore — Runtime schema injection Extends `SchemaStore` to allow adding schemas from in-memory hashes rather than disk files. Useful in tests or when schemas arrive dynamically (e.g., fetched from a registry). ```ruby store = AvroTurf::MutableSchemaStore.new(path: "app/schemas/") store.add_schema( "name" => "event", "namespace" => "analytics", "type" => "record", "fields" => [ { "name" => "event_type", "type" => "string" }, { "name" => "timestamp", "type" => "long" } ] ) avro = AvroTurf.new(schema_store: store) binary = avro.encode( { "event_type" => "page_view", "timestamp" => Time.now.to_i }, schema_name: "event", namespace: "analytics" ) ``` ``` -------------------------------- ### AvroTurf::SchemaStore Source: https://context7.com/dasch/avro_turf/llms.txt A file-backed schema store that lazily loads schemas from disk on first access. It is thread-safe and can be instantiated and injected into `AvroTurf` for custom schema management. ```APIDOC ## AvroTurf::SchemaStore — File-backed schema store Lazily loads schemas from disk on first access. Thread-safe via an internal `Mutex`. Used internally by `AvroTurf` and `Messaging`, but can be instantiated and injected independently. ```ruby store = AvroTurf::SchemaStore.new(path: "app/schemas/") # Lazy load a single schema (and any transitive dependencies) schema = store.find("person", "contacts") # => # # Eagerly load every schema under the path store.load_schemas! # Inject a custom store into AvroTurf avro = AvroTurf.new(schema_store: store) ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#compatibility_issues Source: https://context7.com/dasch/avro_turf/llms.txt Lists compatibility issues between a proposed schema and the existing schema for a subject. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#compatibility_issues ### Description Provides a list of specific issues that prevent a proposed schema from being compatible with the currently registered schema for a given subject. ### Method `compatibility_issues(subject, schema)` ### Parameters * **subject** (String) - The name of the schema subject. * **schema** (Avro::Schema) - The Avro schema object to check. ### Response * **issues** (Array) - An array of strings describing the compatibility issues, or an empty array if the schema is compatible. ### Request Example ```ruby registry.compatibility_issues("greeting", Avro::Schema.parse(schema_json)) # => [] or ["#/fields/0: ..."] ``` ``` -------------------------------- ### Caching Wrapper for Schema Registry Client Source: https://context7.com/dasch/avro_turf/llms.txt Wraps a `ConfluentSchemaRegistry`-compatible client with an in-memory or disk cache to reduce redundant HTTP calls. Useful for performance optimization. ```ruby require "avro_turf/cached_confluent_schema_registry" require "avro_turf/disk_cache" upstream = AvroTurf::ConfluentSchemaRegistry.new("http://schema-registry:8081/") # In-memory cache (default) cached = AvroTurf::CachedConfluentSchemaRegistry.new(upstream) # Persistent disk cache — survives process restarts disk_cache = AvroTurf::DiskCache.new("/var/cache/avro_turf", logger: Logger.new($stdout)) cached = AvroTurf::CachedConfluentSchemaRegistry.new(upstream, cache: disk_cache) # Inject into Messaging require "avro_turf/messaging" avro = AvroTurf::Messaging.new(registry: cached, schemas_path: "app/schemas/") ``` -------------------------------- ### Register Schema Source: https://github.com/dasch/avro_turf/blob/master/README.md Register a schema with the Schema Registry. You can specify the schema name, namespace, and a custom subject under which to register it. ```ruby # Register schema fetched from store by name schema, schema_id = avro.register_schema(schema_name: 'greeting') # Specify namespace (same as schema_name: 'somewhere.greeting') schema, schema_id = avro.register_schema(schema_name: 'greeting', namespace: 'somewhere') # Customize subject under which to register schema schema, schema_id = avro.register_schema(schema_name: 'greeting', namespace: 'somewhere', subject: 'test') ``` -------------------------------- ### AvroTurf::SchemaStore - File-backed Schema Storage Source: https://context7.com/dasch/avro_turf/llms.txt Manages Avro schemas loaded from disk. It lazily loads schemas on first access and is thread-safe. Can be instantiated independently and injected into AvroTurf. ```ruby store = AvroTurf::SchemaStore.new(path: "app/schemas/") # Lazy load a single schema (and any transitive dependencies) schema = store.find("person", "contacts") # => # # Eagerly load every schema under the path store.load_schemas! # Inject a custom store into AvroTurf avro = AvroTurf.new(schema_store: store) ``` -------------------------------- ### Define Inter-Schema References in Avro Source: https://context7.com/dasch/avro_turf/llms.txt Demonstrates how Avro schemas can reference other schemas by name, enabling cross-file type reuse. Schema files must be named correctly and placed in the appropriate namespace sub-directory. ```json // app/schemas/contacts/address.avsc { "name": "address", "namespace": "contacts", "type": "record", "fields": [ { "name": "street", "type": "string" }, { "name": "city", "type": "string" } ] } ``` ```json // app/schemas/contacts/person.avsc — references contacts.address by name { "name": "person", "namespace": "contacts", "type": "record", "fields": [ { "name": "full_name", "type": "string" }, { "name": "address", "type": "contacts.address" } ] } ``` -------------------------------- ### AvroTurf::Messaging#fetch_schema Source: https://context7.com/dasch/avro_turf/llms.txt Fetches a schema from the registry by subject name and an optional version. ```APIDOC ## AvroTurf::Messaging#fetch_schema ### Description Retrieves a schema from the registry by its subject name. If a version is specified, it fetches that specific version; otherwise, it fetches the latest version. ### Method `fetch_schema(subject:, version: nil)` ### Parameters * **subject** (String) - The name of the schema subject. * **version** (Integer, optional) - The specific version of the schema to fetch. ### Response * **schema** (Avro::Schema) - The fetched Avro schema. * **id** (Integer) - The unique ID of the fetched schema. ### Request Example ```ruby schema, id = avro.fetch_schema(subject: "analytics.event") puts "Latest schema id: #{id}" schema_v1, id_v1 = avro.fetch_schema(subject: "analytics.event", version: 1) ``` ``` -------------------------------- ### AvroTurf::Messaging - Schema Registry Integration Source: https://context7.com/dasch/avro_turf/llms.txt Provides compact encoding using a Schema Registry. Requires `require 'avro_turf/messaging'` and a running Confluent Schema Registry. Schemas are registered automatically on first use. ```ruby require "avro_turf/messaging" avro = AvroTurf::Messaging.new( registry_url: "http://schema-registry:8081/", schemas_path: "app/schemas/", namespace: "analytics", logger: Logger.new($stdout) ) # Encode — registers schema on first call, then reuses the cached ID binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, schema_name: "event" ) # Encode by pinning subject + version (no auto-registration) binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, subject: "analytics.event", version: 1 ) # Encode by explicit schema_id binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, schema_id: 42 ) ``` -------------------------------- ### AvroTurf::CachedConfluentSchemaRegistry Source: https://context7.com/dasch/avro_turf/llms.txt Wraps a schema registry client with caching capabilities. ```APIDOC ## AvroTurf::CachedConfluentSchemaRegistry ### Description Provides a caching layer over any `AvroTurf::ConfluentSchemaRegistry`-compatible client. This wrapper reduces redundant HTTP calls by storing frequently accessed schemas in memory or on disk. ### Initialization `CachedConfluentSchemaRegistry.new(upstream_registry, cache: nil)` ### Parameters * **upstream_registry** (AvroTurf::ConfluentSchemaRegistry) - The underlying schema registry client to wrap. * **cache** (Object, optional) - A cache object conforming to the cache interface. Defaults to an in-memory cache. ### Request Example ```ruby require "avro_turf/cached_confluent_schema_registry" require "avro_turf/disk_cache" upstream = AvroTurf::ConfluentSchemaRegistry.new("http://schema-registry:8081/") # In-memory cache (default) cached = AvroTurf::CachedConfluentSchemaRegistry.new(upstream) # Persistent disk cache disk_cache = AvroTurf::DiskCache.new("/var/cache/avro_turf", logger: Logger.new($stdout)) cached = AvroTurf::CachedConfluentSchemaRegistry.new(upstream, cache: disk_cache) # Inject into Messaging require "avro_turf/messaging" avro = AvroTurf::Messaging.new(registry: cached, schemas_path: "app/schemas/") ``` ``` -------------------------------- ### Encode and Decode with Inter-Schema References using AvroTurf Source: https://context7.com/dasch/avro_turf/llms.txt Encodes data using a schema that references another schema, and then decodes it. Requires schemas to be correctly defined and placed. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") binary = avro.encode( { "full_name" => "Jane Doe", "address" => { "street" => "123 Main St", "city" => "Springfield" } }, schema_name: "person", namespace: "contacts" ) avro.decode(binary, schema_name: "person", namespace: "contacts") # => {"full_name"=>"Jane Doe", "address"=>{"street"=>"123 Main St", "city"=>"Springfield"}} ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#compatible? Source: https://context7.com/dasch/avro_turf/llms.txt Checks if a schema is compatible with the existing schema for a given subject. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#compatible? ### Description Determines if a proposed schema is compatible with the currently registered schema for a given subject, according to the registry's compatibility rules. ### Method `compatible?(subject, schema)` ### Parameters * **subject** (String) - The name of the schema subject. * **schema** (Avro::Schema) - The Avro schema object to check for compatibility. ### Response * **compatible** (Boolean or nil) - `true` if compatible, `false` if not, `nil` if compatibility check is not applicable or fails. ### Request Example ```ruby registry.compatible?("greeting", Avro::Schema.parse(schema_json)) # => true / false / nil ``` ``` -------------------------------- ### Fetch Schema by ID Source: https://github.com/dasch/avro_turf/blob/master/README.md Fetch a schema directly from the Schema Registry using its unique schema ID. This is an efficient way to retrieve a schema when its ID is known. ```ruby # Fetch schema by id schema, schema_id = avro.fetch_schema_by_id(3) ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#update_global_config Source: https://context7.com/dasch/avro_turf/llms.txt Updates the global compatibility settings for the schema registry. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#update_global_config ### Description Modifies the default schema compatibility level applied across all subjects in the registry. ### Method `update_global_config(compatibility:)` ### Parameters * **compatibility** (String) - The new global compatibility level (e.g., "FULL", "BACKWARD", "NONE"). ### Request Example ```ruby registry.update_global_config(compatibility: "FULL") ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#subjects Source: https://context7.com/dasch/avro_turf/llms.txt Lists all subjects registered in the schema registry. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#subjects ### Description Retrieves a list of all subject names currently registered in the schema registry. ### Method `subjects` ### Response * **subjects** (Array) - An array of subject names. ### Request Example ```ruby registry.subjects # => ["contacts.person", "analytics.event", ...] ``` ``` -------------------------------- ### Low-Level Confluent Schema Registry Client Source: https://context7.com/dasch/avro_turf/llms.txt Provides a direct HTTP client for the Confluent Schema Registry REST API. Supports authentication, TLS, proxy, and custom DNS. ```ruby require "avro_turf/confluent_schema_registry" registry = AvroTurf::ConfluentSchemaRegistry.new( "https://schema-registry.example.com", user: "api-key", password: "api-secret", ssl_ca_file: "/etc/ssl/registry-ca.pem" ) # List all subjects registry.subjects # => ["contacts.person", "analytics.event", ...] # Register a schema (returns integer id) schema_json = '{"type":"record","name":"greeting","fields":[{"name":"title","type":"string"}]}' id = registry.register("greeting", Avro::Schema.parse(schema_json)) # => 3 # Fetch a schema by id registry.fetch(3) # => '{"type":"record","name":"greeting",...}' # Get all versions for a subject registry.subject_versions("greeting") # => [1, 2] # Get a specific version data = registry.subject_version("greeting", 1) # => {"subject"=>"greeting", "version"=>1, "id"=>3, "schema"=>"..."} # Check schema compatibility registry.compatible?("greeting", Avro::Schema.parse(schema_json)) # => true / false / nil registry.compatibility_issues("greeting", Avro::Schema.parse(schema_json)) # => [] or ["#/fields/0: ...", ...] # Update compatibility settings registry.update_global_config(compatibility: "FULL") registry.update_subject_config("greeting", compatibility: "NONE") ``` -------------------------------- ### Check Schema Compatibility with Confluent Schema Registry Source: https://github.com/dasch/avro_turf/blob/master/README.md Use the ConfluentSchemaRegistry client to check if a schema is compatible with a subject in the registry. Requires AvroTurf and AvroTurf::ConfluentSchemaRegistry. The client can return true, nil, or false based on compatibility. ```ruby require 'avro_turf' require 'avro_turf/confluent_schema_registry' schema = <<-JSON { "name": "person", "type": "record", "fields": [ { "name": "full_name", "type": "string" }, { "name": "address", "type": "address" } ] } JSON registry = AvroTurf::ConfluentSchemaRegistry.new("http://my-registry:8081/") # Returns true if the schema is compatible, nil if the subject or version is not registered, and false if incompatible. registry.compatible?("person", schema) # Returns an array of any breaking changes, nil if the subject or version is not registered registry.compatibility_issues("person", schema) ``` -------------------------------- ### AvroTurf#decode_all Source: https://context7.com/dasch/avro_turf/llms.txt Returns an `Avro::DataFile::Reader` enumerable containing every record encoded in a multi-record Avro binary string. ```APIDOC ## AvroTurf#decode_all — Decode all records from an Avro data file Returns an `Avro::DataFile::Reader` enumerable containing every record encoded in a multi-record Avro binary string. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") # Build a multi-record file stream = StringIO.new avro.encode_to_stream({ "name" => "Alice", "age" => 30 }, schema_name: "person", stream: stream) # (append more records as needed) records = avro.decode_all(stream.string) records.each { |r| puts r["name"] } # => Alice ``` ``` -------------------------------- ### AvroTurf#decode / #decode_first Source: https://context7.com/dasch/avro_turf/llms.txt Decodes the first record from an Avro binary string. An optional `schema_name` can be supplied to use a reader schema different from the writer schema (enabling schema evolution). ```APIDOC ## AvroTurf#decode / #decode_first — Decode Avro data-file format Decodes the first record from an Avro binary string. An optional `schema_name` can be supplied to use a reader schema different from the writer schema (enabling schema evolution). ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") binary = avro.encode({ "name" => "Jane Doe", "age" => 28 }, schema_name: "person") # Decode using the writer schema embedded in the data record = avro.decode(binary) # => { "name" => "Jane Doe", "age" => 28 } # Decode with an explicit reader schema (projection / evolution) record = avro.decode(binary, schema_name: "person_summary", namespace: "contacts") # => { "name" => "Jane Doe" } # only fields present in person_summary ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#register Source: https://context7.com/dasch/avro_turf/llms.txt Registers a new schema with the schema registry. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#register ### Description Registers a given Avro schema with the schema registry under the specified subject. Returns the unique ID assigned to the schema. ### Method `register(subject, schema)` ### Parameters * **subject** (String) - The name of the subject under which to register the schema. * **schema** (Avro::Schema) - The Avro schema object to register. ### Response * **id** (Integer) - The unique ID of the registered schema. ### Request Example ```ruby schema_json = '{"type":"record","name":"greeting","fields":[{"name":"title","type":"string"}]}' schema = Avro::Schema.parse(schema_json) id = registry.register("greeting", schema) # => 3 ``` ``` -------------------------------- ### Fetch Schema from Registry Source: https://github.com/dasch/avro_turf/blob/master/README.md Fetch the latest schema for a given subject from the Schema Registry. You can also fetch a specific version of a schema by providing the version number. ```ruby # You can also work with schema through this interface: # Fetch latest schema for subject from registry schema, schema_id = avro.fetch_schema(subject: 'greeting') # Fetch specific version schema, schema_id = avro.fetch_schema(subject: 'greeting', version: 1) ``` -------------------------------- ### AvroTurf::Messaging#fetch_schema_by_id Source: https://context7.com/dasch/avro_turf/llms.txt Fetches a schema from the registry using its numeric ID. ```APIDOC ## AvroTurf::Messaging#fetch_schema_by_id ### Description Retrieves a schema from the registry using its globally unique integer ID and caches it for future use. ### Method `fetch_schema_by_id(id)` ### Parameters * **id** (Integer) - The unique integer ID of the schema to fetch. ### Response * **schema** (Avro::Schema) - The fetched Avro schema. * **id** (Integer) - The ID of the fetched schema. ### Request Example ```ruby schema, id = avro.fetch_schema_by_id(7) puts schema.to_s # => JSON representation of the Avro schema ``` ``` -------------------------------- ### AvroTurf::Messaging Source: https://context7.com/dasch/avro_turf/llms.txt Provides a compact encoding format using a 5-byte header (magic byte + 4-byte schema ID) instead of embedding the full schema. This requires a running Confluent Schema Registry and automatically registers schemas on first use. ```APIDOC ## AvroTurf::Messaging — Schema Registry wire format Provides compact encoding where a 5-byte header (magic byte + 4-byte schema ID) replaces the full embedded schema. Schemas are registered automatically on first use. Requires `require 'avro_turf/messaging'` and a running Confluent Schema Registry. ```ruby require "avro_turf/messaging" avro = AvroTurf::Messaging.new( registry_url: "http://schema-registry:8081/", schemas_path: "app/schemas/", namespace: "analytics", logger: Logger.new($stdout) ) # Encode — registers schema on first call, then reuses the cached ID binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, schema_name: "event" ) # Encode by pinning subject + version (no auto-registration) binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, subject: "analytics.event", version: 1 ) # Encode by explicit schema_id binary = avro.encode( { "event_type" => "purchase", "timestamp" => 1_700_000_000 }, schema_id: 42 ) ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#subject_versions Source: https://context7.com/dasch/avro_turf/llms.txt Retrieves all registered versions for a given schema subject. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#subject_versions ### Description Gets a list of all schema versions that have been registered under a specific subject. ### Method `subject_versions(subject)` ### Parameters * **subject** (String) - The name of the schema subject. ### Response * **versions** (Array) - An array of schema version numbers. ### Request Example ```ruby registry.subject_versions("greeting") # => [1, 2] ``` ``` -------------------------------- ### Decode First Record from IO Stream with AvroTurf Source: https://context7.com/dasch/avro_turf/llms.txt Decodes the first Avro record from an open IO stream without loading the entire file. Useful for large files. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") File.open("output.avro", "rb") do |file| record = avro.decode_stream(file, schema_name: "person") puts record.inspect # => {"name"=>"Jane Doe", "age"=>28} end ``` -------------------------------- ### Fetch Schema by Numeric ID Source: https://context7.com/dasch/avro_turf/llms.txt Retrieves a schema from the registry using its unique integer ID. The schema is cached after fetching. ```ruby require "avro_turf/messaging" avro = AvroTurf::Messaging.new(registry_url: "http://schema-registry:8081/") schema, id = avro.fetch_schema_by_id(7) puts schema.to_s # => JSON representation of the Avro schema ``` -------------------------------- ### Decode First Record from Avro Data-File Source: https://context7.com/dasch/avro_turf/llms.txt Decode the first Avro record from a binary string. An optional `schema_name` can be provided to use a different reader schema, enabling schema evolution and projection. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") binary = avro.encode({ "name" => "Jane Doe", "age" => 28 }, schema_name: "person") # Decode using the writer schema embedded in the data record = avro.decode(binary) # => { "name" => "Jane Doe", "age" => 28 } # Decode with an explicit reader schema (projection / evolution) record = avro.decode(binary, schema_name: "person_summary", namespace: "contacts") # => { "name" => "Jane Doe" } # only fields present in person_summary ``` -------------------------------- ### AvroTurf#encode / #decode Source: https://context7.com/dasch/avro_turf/llms.txt Encodes data into Avro binary format using a specified schema and namespace, and decodes binary data back into a Ruby hash. Supports inter-schema references by automatically resolving referenced schemas from disk. ```APIDOC ## Inter-schema references — Cross-file type reuse AvroTurf resolves referenced schemas from disk automatically when a schema mentions a type that is not yet loaded. Schema files must be named `.avsc` and placed in the correct namespace sub-directory. ```json // app/schemas/contacts/address.avsc { "name": "address", "namespace": "contacts", "type": "record", "fields": [ { "name": "street", "type": "string" }, { "name": "city", "type": "string" } ] } ``` ```json // app/schemas/contacts/person.avsc — references contacts.address by name { "name": "person", "namespace": "contacts", "type": "record", "fields": [ { "name": "full_name", "type": "string" }, { "name": "address", "type": "contacts.address" } ] } ``` ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") binary = avro.encode( { "full_name" => "Jane Doe", "address" => { "street" => "123 Main St", "city" => "Springfield" } }, schema_name: "person", namespace: "contacts" ) avro.decode(binary, schema_name: "person", namespace: "contacts") # => {"full_name"=>"Jane Doe", "address"=>{"street"=>"123 Main St", "city"=>"Springfield"}} ``` ``` -------------------------------- ### Decode All Records from Avro Data File Source: https://context7.com/dasch/avro_turf/llms.txt Returns an enumerable `Avro::DataFile::Reader` for all records in a multi-record Avro binary string. Useful for processing large files or streams. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") # Build a multi-record file stream = StringIO.new avro.encode_to_stream({ "name" => "Alice", "age" => 30 }, schema_name: "person", stream: stream) # (append more records as needed) records = avro.decode_all(stream.string) records.each { |r| puts r["name"] } # => Alice ``` -------------------------------- ### AvroTurf::Messaging#register_schema Source: https://context7.com/dasch/avro_turf/llms.txt Registers a local Avro schema with the schema registry. ```APIDOC ## AvroTurf::Messaging#register_schema ### Description Pushes a local Avro schema file (`.avsc`) to the schema registry. It returns the registered schema and its unique ID. The subject can be explicitly provided or defaults to the schema's fully-qualified name. ### Method `register_schema(schema_name:, namespace:, subject: nil)` ### Parameters * **schema_name** (String) - The name of the schema file (without extension). * **namespace** (String) - The namespace of the schema. * **subject** (String, optional) - A custom subject name to register the schema under. Defaults to the schema's fully-qualified name. ### Response * **schema** (Avro::Schema) - The registered Avro schema. * **id** (Integer) - The unique ID assigned to the registered schema. ### Request Example ```ruby # Register under the schema's own fullname schema, id = avro.register_schema(schema_name: "event", namespace: "analytics") # Register under a custom subject schema, id = avro.register_schema( schema_name: "event", namespace: "analytics", subject: "my-kafka-topic-value" ) puts "Registered as id #{id}" ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#update_subject_config Source: https://context7.com/dasch/avro_turf/llms.txt Updates the compatibility settings for a specific schema subject. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#update_subject_config ### Description Sets the schema compatibility level for a particular subject, overriding the global configuration if necessary. ### Method `update_subject_config(subject, compatibility:)` ### Parameters * **subject** (String) - The name of the schema subject. * **compatibility** (String) - The new compatibility level for this subject (e.g., "NONE", "BACKWARD"). ### Request Example ```ruby registry.update_subject_config("greeting", compatibility: "NONE") ``` ``` -------------------------------- ### AvroTurf::ConfluentSchemaRegistry#subject_version Source: https://context7.com/dasch/avro_turf/llms.txt Retrieves a specific version of a schema for a given subject. ```APIDOC ## AvroTurf::ConfluentSchemaRegistry#subject_version ### Description Fetches the details of a specific version of a schema registered under a given subject, including the schema content. ### Method `subject_version(subject, version)` ### Parameters * **subject** (String) - The name of the schema subject. * **version** (Integer) - The specific version number of the schema. ### Response * **data** (Hash) - A hash containing schema details, including subject, version, ID, and the schema itself. ### Request Example ```ruby data = registry.subject_version("greeting", 1) # => {"subject"=>"greeting", "version"=>1, "id"=>3, "schema"=>"..."} ``` ``` -------------------------------- ### Validate Data Against Avro Schema with AvroTurf Source: https://context7.com/dasch/avro_turf/llms.txt Checks if data is compatible with a specified Avro schema. Accepts validation options similar to encode_to_stream. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") avro.valid?({ "name" => "Jane", "age" => 28 }, schema_name: "person") # => true avro.valid?({ "name" => "Jane" }, schema_name: "person") # => false (missing age) avro.valid?({ "name" => 42, "age" => 28 }, schema_name: "person") # => false (wrong type) ``` -------------------------------- ### AvroTurf#encode Source: https://context7.com/dasch/avro_turf/llms.txt Encodes a Ruby Hash into an Avro data-file binary string using the named schema. Pass `validate: true` to raise `Avro::SchemaValidator::ValidationError` with a descriptive message when the data does not match the schema. ```APIDOC ## AvroTurf#encode — Encode data to Avro data-file format Encodes a Ruby Hash into an Avro data-file binary string using the named schema. Pass `validate: true` to raise `Avro::SchemaValidator::ValidationError` with a descriptive message when the data does not match the schema. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") # Basic encoding binary = avro.encode( { "name" => "Jane Doe", "age" => 28 }, schema_name: "person" ) # With pre-encoding validation begin binary = avro.encode( { "titl" => "hello, world" }, # typo — wrong field name schema_name: "greeting", validate: true ) rescue Avro::SchemaValidator::ValidationError => e puts e.message # => "The datum {\"titl\"=>...} is not an example of schema ..." end # Skip auto-registration (schema must already exist in registry when used with Messaging) binary = avro.encode( { "name" => "Jane Doe", "age" => 28 }, schema_name: "person", register_schemas: false ) ``` ``` -------------------------------- ### AvroTurf#decode_stream / #decode_first_from_stream Source: https://context7.com/dasch/avro_turf/llms.txt Decodes the first record from an already-open IO stream. This is useful for reading large Avro files without loading the entire file into memory. ```APIDOC ## AvroTurf#decode_stream / #decode_first_from_stream — Decode from IO Decodes the first record from an already-open IO stream. Useful when reading large Avro files without loading them entirely into memory. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") File.open("output.avro", "rb") do |file| record = avro.decode_stream(file, schema_name: "person") puts record.inspect # => {"name"=>"Jane Doe", "age"=>28} end ``` ``` -------------------------------- ### Encode Data with Automatic Schema Registration Source: https://github.com/dasch/avro_turf/blob/master/README.md Encode data using a schema name. If the schema is new, it will be automatically registered with the Schema Registry. This is the simplest way to encode data when the schema is not yet registered. ```ruby # The API for encoding and decoding data is similar to the default one. Encoding # data has the side effect of registering the schema. This only happens the first # time a schema is used. data = avro.encode({ "title" => "hello, world" }, schema_name: "greeting") ``` -------------------------------- ### AvroTurf#valid? Source: https://context7.com/dasch/avro_turf/llms.txt Validates if the provided data is compatible with the named Avro schema. It returns true if compatible, and false otherwise. Accepts the same validation options as `encode_to_stream`. ```APIDOC ## AvroTurf#valid? — Validate data against a schema Returns `true` if the data is compatible with the named Avro schema, `false` otherwise. Accepts the same `validate_options` hash as `encode_to_stream`. ```ruby avro = AvroTurf.new(schemas_path: "app/schemas/") avro.valid?({ "name" => "Jane", "age" => 28 }, schema_name: "person") # => true avro.valid?({ "name" => "Jane" }, schema_name: "person") # => false (missing age) avro.valid?({ "name" => 42, "age" => 28 }, schema_name: "person") # => false (wrong type) ``` ```