### Java Configuration Constants

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/examples/README.md

Required configuration constants that must be updated before running examples. Includes server endpoints for Zerobus and Unity Catalog, table name, and OAuth client credentials. These values are essential for establishing connection and authentication with the Databricks workspace.

```java
private static final String SERVER_ENDPOINT = "your-shard-id.zerobus.region.cloud.databricks.com";
private static final String UNITY_CATALOG_ENDPOINT = "https://your-workspace.cloud.databricks.com";
private static final String TABLE_NAME = "catalog.schema.table";
private static final String CLIENT_ID = "your-oauth-client-id";
private static final String CLIENT_SECRET = "your-oauth-client-secret";
```

--------------------------------

### Protobuf Schema Definition

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/examples/README.md

Defines the AirQuality protocol buffer message schema used by the examples. The schema includes optional fields for device name, temperature, and humidity. Users can generate their own Java classes from custom .proto files using protoc and update examples to use their message types.

```proto
syntax = "proto2";

message AirQuality {
    optional string device_name = 1;
    optional int32 temp = 2;
    optional int64 humidity = 3;
}
```

```bash
protoc --java_out=. your_schema.proto
```

--------------------------------

### Build Project with Maven

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

Executes Maven clean install to generate protobuf classes, compile source code, run tests, and install artifacts to the local Maven repository. This command performs a full build cycle. Requires Maven 3.6+ and Java 8+ with proper environment configuration.

```bash
mvn clean install
```

--------------------------------

### Create Maven Project for Zerobus SDK (Bash)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Commands to create a new directory for a Maven project and navigate into it. This is the initial setup step for a Java application using the Zerobus SDK.

```bash
mkdir my-zerobus-app
cd my-zerobus-app
```

--------------------------------

### Clone Zerobus SDK Repository

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

Clones the Zerobus SDK repository from GitHub and changes into the project directory. This is the first step in setting up the development environment. Requires Git to be installed on your system.

```bash
git clone https://github.com/databricks/zerobus-sdk-java.git
cd zerobus-sdk-java
```

--------------------------------

### Generate Proto File for a Simple Table

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

Example of generating a proto file for a 'users' table using the GenerateProto tool. It demonstrates both running from the SDK JAR and from the source repository script. The output shows a basic proto message definition with mapped data types.

```bash
java -jar databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://myworkspace.cloud.databricks.com" \
  --client-id "abc123" \
  --client-secret "secret123" \
  --table "my_catalog.my_schema.users" \
  --output "users.proto"
```

```bash
./tools/generate_proto.sh \
  --uc-endpoint "https://myworkspace.cloud.databricks.com" \
  --client-id "abc123" \
  --client-secret "secret123" \
  --table "my_catalog.my_schema.users" \
  --output "users.proto"
```

```protobuf
syntax = "proto2";

message users {
    required int32 user_id = 1;
    required string username = 2;
    optional string email = 3;
    required int64 created_at = 4;
}
```

--------------------------------

### Java Blocking Ingestion Example

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/examples/README.md

Demonstrates synchronous record ingestion where each record is waited for before proceeding. Best for low-volume ingestion (< 1000 records/sec) and critical data requiring immediate confirmation. Waits for each record to be durably written with simple error handling and predictable behavior.

```java
javac -cp "../target/databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar" \
  src/main/java/com/databricks/zerobus/examples/BlockingIngestionExample.java

java -cp "../target/databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar:src/main/java" \
  com.databricks.zerobus.examples.BlockingIngestionExample
```

--------------------------------

### Get StreamConfigurationOptions Builder in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Shows the static method to get a builder for creating custom `StreamConfigurationOptions`.

```java
static StreamConfigurationOptionsBuilder builder()
```

--------------------------------

### Executing Maven Build Commands in Bash

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

This set of bash commands handles Maven operations for cleaning, compiling, testing, formatting, packaging, installing, and generating protobuf classes in the Java SDK project. Use spotless for code formatting checks and applies; package creates both regular and fat JARs. Requires Maven and protobuf plugin; runs on Ubuntu and Windows for CI compatibility.

```bash
# Clean build
mvn clean

# Compile code
mvn compile

# Run tests
mvn test

# Format code
mvn spotless:apply

# Check formatting
mvn spotless:check

# Create JARs (regular + fat JAR)
mvn package

# Install to local Maven repo
mvn install

# Generate protobuf classes
mvn protobuf:compile
```

--------------------------------

### Java Non-Blocking Ingestion Example

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/examples/README.md

Demonstrates asynchronous record ingestion for maximum throughput using CompletableFutures. Best for high-volume ingestion (> 10,000 records/sec), batch processing, and stream processing applications. Features automatic buffering, flow control, ack callbacks for progress tracking, and batch flush capability.

```java
javac -cp "../target/databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar" \
  src/main/java/com/databricks/zerobus/examples/NonBlockingIngestionExample.java

java -cp "../target/databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar:src/main/java" \
  com.databricks.zerobus.examples.NonBlockingIngestionExample
```

--------------------------------

### Maven Dependency for slf4j-simple

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Adds the slf4j-simple library as a Maven dependency, which is the simplest SLF4J implementation for getting started with logging. This allows the Zerobus SDK, which depends on slf4j-api, to output log messages to the console.

```xml
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-simple</artifactId>
    <version>1.7.36</version>
</dependency>
```

--------------------------------

### Compile Protocol Buffer Schema using protoc (Bash)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Command to compile a Protocol Buffer schema file (`.proto`) into Java source code using the `protoc` compiler. Ensure `protoc` is installed and accessible in your PATH.

```bash
protoc --java_out=src/main/java src/main/proto/record.proto
```

--------------------------------

### Writing Java Unit Tests with JUnit 5

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

This example shows a unit test for record ingestion functionality using JUnit 5 and Mockito for mocking. It follows the Given-When-Then structure to verify that ingestion completes successfully and the stream state remains OPENED. Tests must be fast (under 1 second), use descriptive names, and handle exceptions; dependencies include JUnit 5 and Mockito, with timeouts for async operations.

```java
@Test
public void testSingleRecordIngestAndAcknowledgment() throws Exception {
    // Given
    ZerobusStream<TestMessage> stream = createTestStream();

    // When
    CompletableFuture<Void> result = stream.ingestRecord(testMessage);

    // Then
    result.get(5, TimeUnit.SECONDS);
    assertEquals(StreamState.OPENED, stream.getState());
}
```

--------------------------------

### Generate Proto with Custom Message Name

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

This example shows how to specify a custom message name for the generated proto file using the `--proto-msg` argument. It's useful when you want a different message name than the table name. The example uses the SDK JAR execution method.

```bash
java -jar databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://myworkspace.cloud.databricks.com" \
  --client-id "abc123" \
  --client-secret "secret123" \
  --table "my_catalog.my_schema.events" \
  --output "events.proto" \
  --proto-msg "EventRecord"
```

--------------------------------

### Get Default StreamConfigurationOptions in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Provides the static method to obtain the default `StreamConfigurationOptions`.

```java
static StreamConfigurationOptions getDefault()
```

--------------------------------

### Bash Performance Tuning Commands

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/examples/README.md

JVM memory configuration command to prevent Out of Memory errors during high-volume non-blocking ingestion. The -Xmx4g flag sets maximum heap size to 4GB, which is recommended for applications handling large volumes of concurrent records and buffering.

```bash
java -Xmx4g -cp ... com.databricks.zerobus.examples.NonBlockingIngestionExample
```

--------------------------------

### Logback Configuration for Zerobus SDK

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Defines a Logback configuration in `logback.xml` to route logs from the com.databricks.zerobus package to the console at DEBUG level. This setup ensures detailed logging for the Zerobus SDK while other logs remain at INFO level.

```xml
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <logger name="com.databricks.zerobus" level="DEBUG"/>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>
```

--------------------------------

### Running Maven Tests in Bash

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

These bash commands execute Maven tests for the Java project. They support running all tests, specific classes, or individual methods using the -Dtest flag. Requires Maven installation and works on Java 11, 17, or 21; outputs test results and failures for CI validation.

```bash
# Run all tests
mvn test

# Run specific test class
mvn test -Dtest=ZerobusSdkTest

# Run specific test method
mvn test -Dtest=ZerobusSdkTest#testSingleRecordIngestAndAcknowledgment
```

--------------------------------

### Get Zerobus Stream Configuration Options in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Provides the method to access the `StreamConfigurationOptions` used by an active Zerobus stream.

```java
StreamConfigurationOptions getOptions()
```

--------------------------------

### Get Default Instance from TableProperties in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Details the method for retrieving the Protobuf message default instance from a `TableProperties` object.

```java
Message getDefaultInstance()
```

--------------------------------

### Get Table Name from TableProperties in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Shows the method to retrieve the table name from a `TableProperties` object.

```java
String getTableName()
```

--------------------------------

### Blocking Ingestion Example in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Demonstrates synchronous data ingestion using the Zerobus SDK. Each record is ingested and its durability is waited upon before proceeding. This method is suitable for scenarios where immediate feedback on each record is necessary. It requires creating a ZerobusStream and iteratively ingesting records, ensuring stream closure in a finally block.

```java
ZerobusStream<AirQuality> stream = sdk.createStream(
    tableProperties,
    clientId,
    clientSecret
).join();

try {
    for (int i = 0; i < 1000; i++) {
        AirQuality record = AirQuality.newBuilder()
            .setDeviceName("sensor-" + i)
            .setTemp(20 + i % 15)
            .setHumidity(50 + i % 40)
            .build();

        stream.ingestRecord(record).join(); // Wait for durability
    }
} finally {
    stream.close();
}
```

--------------------------------

### Generated Protobuf Schema for Complex Types

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

Example of a generated Protobuf message definition based on a Unity Catalog table with 'tags ARRAY<STRING>' and 'attributes MAP<STRING, STRING>' columns. This demonstrates how complex types are represented in the .proto file.

```protobuf
syntax = "proto2";

message products {
    required int32 product_id = 1;
    required string name = 2;
    repeated string tags = 3;
    map<string, string> attributes = 4;
}
```

--------------------------------

### Non-Blocking Ingestion Example in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Illustrates high-throughput asynchronous data ingestion with the Zerobus SDK. Records are sent without waiting for individual acknowledgments, improving performance. It utilizes StreamConfigurationOptions to set parameters like maxInflightRecords and an ackCallback for processing acknowledgments. Futures are collected to await the completion of all ingestion operations.

```java
StreamConfigurationOptions options = StreamConfigurationOptions.builder()
    .setMaxInflightRecords(50000)
    .setAckCallback(response ->
        System.out.println("Acknowledged offset: " +
            response.getDurabilityAckUpToOffset()))
    .build();

ZerobusStream<AirQuality> stream = sdk.createStream(
    tableProperties,
    clientId,
    clientSecret,
    options
).join();

List<CompletableFuture<Void>> futures = new ArrayList<>();

try {
    for (int i = 0; i < 100000; i++) {
        AirQuality record = AirQuality.newBuilder()
            .setDeviceName("sensor-" + (i % 10))
            .setTemp(20 + i % 15)
            .setHumidity(50 + i % 40)
            .build();

        futures.add(stream.ingestRecord(record));
    }

    // Flush and wait for all records
    stream.flush();
    CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
} finally {
    stream.close();
}
```

--------------------------------

### Get Zerobus Stream Table Properties in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Details the method for retrieving the `TableProperties` associated with an active Zerobus stream.

```java
TableProperties<RecordType> getTableProperties()
```

--------------------------------

### Monitor Zerobus Stream State and Progress

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

This Java example shows how to monitor the lifecycle state and ingestion progress of a Zerobus stream. It utilizes a callback for acknowledged records to track progress and logs the stream's state at various points. Requires the Zerobus SDK.

```java
import com.databricks.zerobus.*;
import java.util.concurrent.atomic.AtomicLong;

AtomicLong ackedRecords = new AtomicLong(0);
AtomicLong latestOffset = new AtomicLong(-1);

StreamConfigurationOptions options = StreamConfigurationOptions.builder()
    .setMaxInflightRecords(50000)
    .setAckCallback(response -> {
        long offset = response.getDurabilityAckUpToOffset();
        latestOffset.set(offset);
        ackedRecords.incrementAndGet();

        // Log progress
        if (ackedRecords.get() % 1000 == 0) {
            System.out.println("Progress: " + ackedRecords.get() +
                             " records acknowledged (offset: " + offset + ")");
        }
    })
    .build();

ZerobusStream<MyRecord> stream = sdk.createStream(
    tableProperties,
    clientId,
    clientSecret,
    options
).join();

// Check initial state
System.out.println("Stream ID: " + stream.getStreamId());
System.out.println("Stream State: " + stream.getState());
// Output: Stream State: OPENED

// Monitor during ingestion
for (int i = 0; i < 10000; i++) {
    stream.ingestRecord(createRecord(i));
}

// Check state during flush
stream.flush();
System.out.println("Stream State: " + stream.getState());
// Output: Stream State: OPENED

// Final statistics
System.out.println("Total acknowledged: " + ackedRecords.get());
System.out.println("Latest offset: " + latestOffset.get());

stream.close();
System.out.println("Stream State: " + stream.getState());
// Output: Stream State: CLOSED
```

--------------------------------

### Get Zerobus Stream ID in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Shows how to obtain the unique stream identifier assigned by the Zerobus server for an active stream.

```java
String getStreamId()
```

--------------------------------

### Get Zerobus Stream State in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Provides the method to retrieve the current state of an active Zerobus stream. Possible states include UNINITIALIZED, OPENED, FLUSHING, RECOVERING, CLOSED, and FAILED.

```java
StreamState getState()
```

--------------------------------

### Handle Zerobus Exceptions with Retry Logic (Java)

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

This example illustrates how to handle `ZerobusException`s during record ingestion, differentiating between retriable and non-retriable errors. It shows a pattern for catching specific exceptions, implementing retry logic with backoff for retriable errors, and re-throwing or exiting for non-retriable errors. The code also includes general error handling for stream creation and a `finally` block to ensure the stream is closed.

```java
import com.databricks.zerobus.*;

ZerobusSdk sdk = new ZerobusSdk(serverEndpoint, workspaceUrl);
TableProperties<MyRecord> tableProperties = new TableProperties<>(
    "main.default.my_table",
    MyRecord.getDefaultInstance()
);

try {
    ZerobusStream<MyRecord> stream = sdk.createStream(
        tableProperties,
        clientId,
        clientSecret
    ).join();

    try {
        for (MyRecord record : recordsToIngest) {
            try {
                stream.ingestRecord(record).join();
            } catch (ZerobusException e) {
                if (e instanceof NonRetriableException) {
                    // Fatal error - do not retry
                    System.err.println("Non-retriable error: " + e.getMessage());
                    // Check if credentials are invalid or table is missing
                    throw e;
                } else {
                    // Retriable error - implement exponential backoff
                    System.warn("Retriable error, retrying: " + e.getMessage());
                    Thread.sleep(2000);
                    stream.ingestRecord(record).join(); // Retry once
                }
            }
        }
    } finally {
        stream.close();
    }

} catch (NonRetriableException e) {
    System.err.println("Fatal stream error: " + e.getMessage());
    // Examples: INVALID_ARGUMENT, NOT_FOUND, UNAUTHENTICATED, OUT_OF_RANGE
    System.exit(1);
} catch (ZerobusException e) {
    System.err.println("Stream error (potentially retriable): " + e.getMessage());
    // Examples: Network issues, temporary server errors
}
```

--------------------------------

### Recover Zerobus Stream After Failure (Java)

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

This code demonstrates how to handle a `ZerobusException` during record ingestion, check if the stream has entered a `FAILED` state, and then use `sdk.recreateStream()` to recover the stream. The `recreateStream` method automatically handles re-ingesting unacknowledged records from the point of failure, allowing ingestion to resume. The example includes closing the recovered stream in a `finally` block.

```java
import com.databricks.zerobus.*;

ZerobusSdk sdk = new ZerobusSdk(serverEndpoint, workspaceUrl);
TableProperties<MyRecord> tableProperties = new TableProperties<>(
    tableName,
    MyRecord.getDefaultInstance()
);

ZerobusStream<MyRecord> stream = sdk.createStream(
    tableProperties,
    clientId,
    clientSecret
).join();

try {
    // Ingest records
    for (MyRecord record : records) {
        stream.ingestRecord(record);
    }
    stream.flush();
} catch (ZerobusException e) {
    System.err.println("Stream failed: " + e.getMessage());

    // Check stream state
    if (stream.getState() == StreamState.FAILED) {
        System.out.println("Attempting to recreate stream...");

        // Recreate stream - automatically re-ingests unacknowledged records
        ZerobusStream<MyRecord> recoveredStream = sdk.recreateStream(stream).join();

        try {
            // Continue with remaining records
            recoveredStream.flush();
            System.out.println("Stream recovered successfully");
        } finally {
            recoveredStream.close();
        }
    }
}
```

--------------------------------

### Build SDK JAR and Run GenerateProto Script

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

This script is recommended for development. It first builds the Zerobus SDK JAR using Maven and then executes the `generate_proto.sh` script to create a .proto file from a Unity Catalog table schema. It requires the Unity Catalog endpoint, client credentials, table name, and output file path.

```bash
# First, build the SDK JAR
mvn package

# Then run the tool
./tools/generate_proto.sh \
  --uc-endpoint "https://your-workspace.cloud.databricks.com" \
  --client-id "your-client-id" \
  --client-secret "your-client-secret" \
  --table "catalog.schema.table_name" \
  --output "output.proto" \
  --proto-msg "TableMessage"
```

--------------------------------

### Run GenerateProto from SDK JAR using -jar

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

This is the simplest method for users with the SDK JAR. It leverages the `Main-Class` manifest entry in the JAR to execute the `GenerateProto` tool directly using the `java -jar` command. This requires only the path to the JAR file and the command-line arguments for Unity Catalog connection and table details.

```java
# Even simpler - just use -jar flag
java -jar databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://your-workspace.cloud.databricks.com" \
  --client-id "your-client-id" \
  --client-secret "your-client-secret" \
  --table "catalog.schema.table_name" \
  --output "output.proto" \
  --proto-msg "TableMessage"
```

--------------------------------

### Run GenerateProto from SDK JAR using -cp

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

This method is recommended for users who have downloaded the SDK JAR without the source code. It executes the `GenerateProto` tool directly from the shaded JAR using the `java -cp` command. This requires specifying the full classpath to the JAR and the main class, along with the necessary arguments for Unity Catalog connection and table details.

```java
# Using the shaded JAR (includes all dependencies)
java -cp databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  com.databricks.zerobus.tools.GenerateProto \
  --uc-endpoint "https://your-workspace.cloud.databricks.com" \
  --client-id "your-client-id" \
  --client-secret "your-client-secret" \
  --table "catalog.schema.table_name" \
  --output "output.proto" \
  --proto-msg "TableMessage"
```

--------------------------------

### Initialize ZerobusSdk in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Shows the constructor for the ZerobusSdk class, which is the main entry point for the SDK. It requires server and Unity Catalog endpoint strings.

```java
ZerobusSdk(String serverEndpoint, String unityCatalogEndpoint)
```

--------------------------------

### Format Code with Spotless

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

Applies Google Java Format to source code and checks formatting compliance using Spotless Maven plugin. Formats Java code, organizes imports, removes unused imports, and sorts Maven POM dependencies. Ensures consistent code style across contributions.

```bash
mvn spotless:apply
```

```bash
mvn spotless:check
```

--------------------------------

### Create Project Structure for Java and Proto Files (Bash)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Bash commands to create the necessary directory structure for Java source files and Protocol Buffer definition files within a Maven project. This organizes the project according to standard conventions.

```bash
mkdir -p src/main/java/com/example
mkdir -p src/main/proto
```

--------------------------------

### Compile Protobuf and Integrate with Zerobus SDK (Bash & Java)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

Steps to compile the generated .proto file using the protobuf compiler and then use the generated Java classes with the Zerobus SDK. This involves placing the proto file, compiling it, and creating a ZerobusStream.

```bash
# Download the SDK JAR (or build it with mvn package)
# Then simply run:
java -jar databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "..." \
  --client-id "..." \
  --client-secret "..." \
  --table "..." \
  --output "output.proto"
```

```bash
protoc --java_out=src/main/java your_proto_file.proto
```

```java
TableProperties<YourMessage> tableProperties = 
    new TableProperties<>("catalog.schema.table", YourMessage.getDefaultInstance());

ZerobusStream<YourMessage> stream = sdk.createStream(
    tableProperties, clientId, clientSecret).join();
```

--------------------------------

### Initialize Zerobus SDK and Create Stream (Java)

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

Initializes the ZerobusSdk with server endpoint and workspace URL, defines table properties using a protobuf message type, and creates a streaming connection to a Delta table with OAuth authentication. Requires Databricks Zerobus SDK dependencies and protobuf definitions for the record type. Outputs a stream ID upon successful creation; no inputs beyond configuration parameters.

```java
import com.databricks.zerobus.*;
import com.example.proto.Record.AirQuality;

// Initialize SDK with server endpoint and workspace URL
ZerobusSdk sdk = new ZerobusSdk(
    "1234567890123456.zerobus.us-west-2.cloud.databricks.com",
    "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com"
);

// Define table properties with protobuf message type
TableProperties<AirQuality> tableProperties = new TableProperties<>(
    "main.default.air_quality",
    AirQuality.getDefaultInstance()
);

// Create stream with default configuration
ZerobusStream<AirQuality> stream = sdk.createStream(
    tableProperties,
    "your-service-principal-application-id",
    "your-service-principal-secret"
).join();

System.out.println("Stream created: " + stream.getStreamId());
// Output: Stream created: abc-123-def-456
```

--------------------------------

### Run Zerobus Client

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

This command executes the ZerobusClient from the compiled Java code, typically used for initiating data ingestion. It requires the 'lib/*' directory and the 'out' directory to be on the classpath. The expected output indicates successful record ingestion.

```bash
java -cp "lib/*:out" com.example.ZerobusClient
```

--------------------------------

### Build and Run Java Client with Maven

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Commands to compile the generated protobuf classes, build the project with Maven, and execute the Zerobus client. Includes alternative steps for creating a standalone JAR and compiling without Maven.

```bash
# First, compile the proto file to generate Java classes
protoc --java_out=src/main/java src/main/proto/record.proto

# Compile and run using Maven
mvn compile
mvn exec:java -Dexec.mainClass="com.example.ZerobusClient"
```

```bash
# Generate proto classes
protoc --java_out=src/main/java src/main/proto/record.proto

# Package into executable JAR (ensure maven-shade-plugin is configured)
mvn package

# Run the JAR
java -jar target/my-zerobus-app-1.0-SNAPSHOT.jar
```

```bash
# Compile without Maven (using classpath with required libs)
javac -cp "lib/*" -d out src/main/java/com/example/ZerobusClient.java src/main/java/com/example/proto/Record.java
```

--------------------------------

### Build Zerobus SDK from Source using Maven

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Commands to clone the Zerobus SDK Java repository and build it locally using Maven. This process generates regular and fat JAR files in the target directory.

```bash
git clone https://github.com/databricks/zerobus-sdk-java.git
cd zerobus-sdk-java
mvn clean package
```

--------------------------------

### Create Zerobus Stream with Default Options in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Demonstrates creating a new ingestion stream with default configuration settings using the ZerobusSdk. This method also returns a CompletableFuture for stream readiness.

```java
<RecordType extends Message> CompletableFuture<ZerobusStream<RecordType>> createStream(
    TableProperties<RecordType> tableProperties,
    String clientId,
    String clientSecret
)
```

--------------------------------

### Generate Protobuf from Unity Catalog Table (Bash)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/tools/README.md

Command to execute the GenerateProto tool from its JAR file. It takes Unity Catalog endpoint, client credentials, table name, and an output file path as input to generate a .proto file. This handles complex types like arrays and maps.

```bash
java -jar databricks-zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://myworkspace.cloud.databricks.com" \
  --client-id "abc123" \
  --client-secret "secret123" \
  --table "my_catalog.my_schema.products" \
  --output "products.proto"
```

--------------------------------

### Run Zerobus Ingest SDK JAR to Generate Protobuf

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Executes the Zerobus ingest SDK JAR with Unity Catalog endpoint, service principal credentials, table name, and output paths. Generates a protobuf definition file for the specified table. Requires Java runtime and access to the target Databricks workspace.

```bash
java -jar zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" \
  --client-id "your-service-principal-application-id" \
  --client-secret "your-service-principal-secret" \
  --table "main.default.air_quality" \
  --output "src/main/proto/record.proto" \
  --proto-msg "AirQuality"
```

--------------------------------

### Java Client to Ingest Records via Zerobus SDK

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Sample Java program that initializes the Zerobus SDK, creates a stream for the AirQuality protobuf message, and ingests 100 synthetic records. Demonstrates configuration of endpoint, credentials, and proper stream closure.

```java
package com.example;

import com.databricks.zerobus.*;
import com.example.proto.Record.AirQuality;

public class ZerobusClient {
    public static void main(String[] args) throws Exception {
        // Configuration
        String serverEndpoint = "1234567890123456.zerobus.us-west-2.cloud.databricks.com";
        String workspaceUrl = "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com";
        String tableName = "main.default.air_quality";
        String clientId = "your-service-principal-application-id";
        String clientSecret = "your-service-principal-secret";

        // Initialize SDK
        ZerobusSdk sdk = new ZerobusSdk(serverEndpoint, workspaceUrl);

        // Configure table properties
        TableProperties<AirQuality> tableProperties = new TableProperties<>(
            tableName,
            AirQuality.getDefaultInstance()
        );

        // Create stream
        ZerobusStream<AirQuality> stream = sdk.createStream(
            tableProperties,
            clientId,
            clientSecret
        ).join();

        try {
            // Ingest records
            for (int i = 0; i < 100; i++) {
                AirQuality record = AirQuality.newBuilder()
                    .setDeviceName("sensor-" + (i % 10))
                    .setTemp(20 + (i % 15))
                    .setHumidity(50 + (i % 40))
                    .build();

                stream.ingestRecord(record).join(); // Wait for durability

                System.out.println("Ingested record " + (i + 1));
            }

            System.out.println("Successfully ingested 100 records!");
        } finally {
            stream.close();
        }
    }
}
```

--------------------------------

### Download Zerobus SDK Fat JAR (Bash)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Bash command to download the Zerobus SDK with all its dependencies included as a fat JAR from Maven Central. This JAR is required for using the Proto Generation Tool.

```bash
# Download from Maven Central
wget https://repo1.maven.org/maven2/com/databricks/zerobus-ingest-sdk/0.1.0/zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar

# Or if you built from source, it's in target/
```

--------------------------------

### Execute Maven Tests

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

Runs the project test suite using Maven to validate functionality and ensure code quality. This command executes all unit and integration tests. Used to verify changes before committing and pushing.

```bash
mvn test
```

--------------------------------

### Log4j 2 Configuration for Zerobus SDK

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Sets up Log4j 2 logging by creating a `log4j2.xml` file. This configuration directs logs from the com.databricks.zerobus package to the console at DEBUG level, while the root logger remains at INFO level.

```xml
<Configuration>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="com.databricks.zerobus" level="debug"/>
        <Root level="info">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>
```

--------------------------------

### Maven pom.xml Configuration for Zerobus SDK (XML)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

An XML configuration file for a Maven project, specifying Java version, encoding, and including essential dependencies like the Zerobus SDK and Google Protocol Buffers. Ensure the versions align with your project requirements.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                             http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>my-zerobus-app</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Zerobus SDK -->
        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>zerobus-ingest-sdk</artifactId>
            <version>0.1.0</version>
        </dependency>

        <!-- Required dependencies (see above for full list) -->
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>3.24.0</version>
        </dependency>
        <!-- Add other dependencies from the list above -->
    </dependencies>
</project>
```

--------------------------------

### Add All Required Dependencies for Zerobus SDK in Maven

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

A comprehensive list of dependencies required for the Zerobus Ingest SDK, including protobuf, gRPC, and SLF4J, to be added to a Maven project's pom.xml. These are necessary when not using the fat JAR.

```xml
<!-- Add these dependencies in addition to the SDK -->
<dependencies>
    <!-- Zerobus SDK -->
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>zerobus-ingest-sdk</artifactId>
        <version>0.1.0</version>
    </dependency>

    <!-- Required dependencies -->
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>3.24.0</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty-shaded</artifactId>
        <version>1.58.0</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>1.58.0</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.58.0</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.36</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.36</version>
    </dependency>
    <dependency>
        <groupId>javax.annotation</groupId>
        <artifactId>javax.annotation-api</artifactId>
        <version>1.3.2</version>
    </dependency>
</dependencies>
```

--------------------------------

### Create Zerobus Stream with Options in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Illustrates how to create a new ingestion stream with custom configuration options using the ZerobusSdk. This method returns a CompletableFuture that resolves when the stream is ready.

```java
<RecordType extends Message> CompletableFuture<ZerobusStream<RecordType>> createStream(
    TableProperties<RecordType> tableProperties,
    String clientId,
    String clientSecret,
    StreamConfigurationOptions options
)
```

--------------------------------

### Documenting Java API with Javadoc

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

This snippet demonstrates adding Javadoc comments to public methods in the Java SDK. It includes standard tags like @param, @return, and @throws to describe parameters, return values, and exceptions. The method ingests records asynchronously using CompletableFuture, with potential blocking on high in-flight records; it requires valid stream state and throws ZerobusException on errors.

```java
/**
 * Ingests a single record into the stream.
 *
 * <p>Returns a CompletableFuture that completes when the record is durably written to storage.
 * This method may block if the maximum number of in-flight records has been reached.
 *
 * @param record The protobuf message to ingest
 * @return A CompletableFuture that completes when the record is acknowledged
 * @throws ZerobusException if the stream is not in a valid state for ingestion
 */
public CompletableFuture<Void> ingestRecord(RecordType record) throws ZerobusException {
    // ...
}
```

--------------------------------

### Create TableProperties for Zerobus Stream in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Illustrates the constructor for `TableProperties`, which is used to configure the target table for a Zerobus stream. It requires the fully qualified table name and a Protobuf message default instance.

```java
TableProperties(String tableName, RecordType defaultInstance)
```

--------------------------------

### Git Feature Branch Workflow

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/CONTRIBUTING.md

Complete sequence of Git commands for feature development workflow. Creates a feature branch, stages changes, commits with sign-off, and pushes to remote repository. Follows the project's pull request process for contributing changes.

```bash
git checkout -b feature/your-feature-name
```

```bash
git add .
```

```bash
git commit -s -m "Add feature: description of your changes"
```

```bash
git push origin feature/your-feature-name
```

--------------------------------

### Maven Dependency for Logback

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Includes the logback-classic library in Maven dependencies, recommended for production logging with the Zerobus SDK. Logback provides a robust and configurable logging framework that integrates seamlessly with SLF4J.

```xml
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.11</version>
</dependency>
```

--------------------------------

### Create Delta Table for Ingestion

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

SQL statement to create a Delta table named 'air_quality' with specified columns: device_name (STRING), temp (INT), and humidity (BIGINT). This table will receive data ingested via the Zerobus SDK. Replace `<catalog_name>` with your actual catalog.

```sql
CREATE TABLE <catalog_name>.default.air_quality (
    device_name STRING,
    temp INT,
    humidity BIGINT
)
USING DELTA;
```

--------------------------------

### Define Unity Catalog Table Schema

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

SQL statement that creates a Delta table with columns for device name, temperature, and humidity. Serves as the source schema from which the SDK generates the protobuf message. Execute in a Databricks SQL notebook or cluster.

```sql
CREATE TABLE main.default.air_quality (
    device_name STRING,
    temp INT,
    humidity BIGINT
)
USING DELTA;
```

--------------------------------

### Generate Protobuf Schema from Unity Catalog Table

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

This bash script uses a Zerobus ingest SDK JAR to generate a Protocol Buffer schema from an existing Delta table in Unity Catalog. It requires the JAR file, Unity Catalog endpoint, credentials, table name, and output path for the proto file. Optionally specifies the Protobuf message name.

```bash
# Download the fat JAR with all dependencies
wget https://repo1.maven.org/maven2/com/databricks/zerobus-ingest-sdk/0.1.0/zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar

# Run the proto generation tool
java -jar zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \
  --uc-endpoint "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" \
  --client-id "your-service-principal-application-id" \
  --client-secret "your-service-principal-secret" \
  --table "main.default.air_quality" \
  --output "src/main/proto/record.proto" \
  --proto-msg "AirQuality"
```

--------------------------------

### Define Protocol Buffer Schema

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Defines a sample schema for air quality data using Protocol Buffers. This schema is used by the Zerobus SDK to structure and validate ingested data. Ensure the generated Java classes match this schema for successful ingestion.

```protobuf
syntax = "proto3";

package com.databricks.zerobus.ingest.example;

option java_package = "com.databricks.zerobus.ingest.example";
option java_multiple_files = true;

message AirQuality {
  string device_name = 1;
  int32 temp = 2;
  int64 humidity = 3;
}
```

--------------------------------

### Include Zerobus SDK and all Dependencies in Gradle (Fat JAR)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Gradle dependency declaration for including the Zerobus Ingest SDK with all its dependencies bundled into a single artifact. This simplifies deployment for standalone applications.

```groovy
dependencies {
    implementation 'com.databricks:zerobus-ingest-sdk:0.1.0:jar-with-dependencies'
}
```

--------------------------------

### Blocking Ingestion with Durability Guarantee (Java)

Source: https://context7.com/databricks/zerobus-sdk-java/llms.txt

Demonstrates synchronous ingestion of records into a Zerobus stream, waiting for each record to be durably written before proceeding. Utilizes the Zerobus SDK for stream creation and record building with protobuf. Takes client credentials and a list of records as inputs; ensures durability but may have lower throughput due to blocking.

```java
import com.databricks.zerobus.*;
import com.example.proto.Record.AirQuality;

ZerobusSdk sdk = new ZerobusSdk(serverEndpoint, workspaceUrl);
TableProperties<AirQuality> tableProperties = new TableProperties<>(
    "main.default.air_quality",
    AirQuality.getDefaultInstance()
);

ZerobusStream<AirQuality> stream = sdk.createStream(
    tableProperties,
    clientId,
    clientSecret
).join();

try {
    for (int i = 0; i < 1000; i++) {
        AirQuality record = AirQuality.newBuilder()
            .setDeviceName("sensor-" + i)
            .setTemp(20 + (i % 15))
            .setHumidity(50 + (i % 40))
            .build();

        // Wait for durability acknowledgment before proceeding
        stream.ingestRecord(record).join();

        if ((i + 1) % 100 == 0) {
            System.out.println("Ingested " + (i + 1) + " records");
        }
    }

    System.out.println("All 1000 records ingested successfully");
} catch (ZerobusException e) {
    System.err.println("Ingestion failed: " + e.getMessage());
    throw e;
} finally {
    stream.close();
}
```

--------------------------------

### Handle Zerobus SDK Exceptions in Java

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Demonstrates how to handle retriable and non-retriable exceptions thrown by the Zerobus SDK. It includes specific catch blocks for NonRetriableException and ZerobusException, suggesting appropriate logging and retry strategies.

```java
try {
    stream.ingestRecord(record);
} catch (NonRetriableException e) {
    // Fatal error - do not retry
    logger.error("Non-retriable error: " + e.getMessage());
    throw e;
} catch (ZerobusException e) {
    // Retriable error - can retry with backoff
    logger.warn("Retriable error: " + e.getMessage());
    // Implement retry logic
}
```

--------------------------------

### Include Zerobus SDK and all Dependencies in Maven (Fat JAR)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

Maven dependency configuration to include the Zerobus Ingest SDK along with all its transitive dependencies in a single JAR. This is useful for creating self-contained applications.

```xml
<dependencies>
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>zerobus-ingest-sdk</artifactId>
        <version>0.1.0</version>
        <classifier>jar-with-dependencies</classifier>
    </dependency>
</dependencies>
```

--------------------------------

### Define Protocol Buffer Schema (Protobuf)

Source: https://github.com/databricks/zerobus-sdk-java/blob/main/README.md

A Protocol Buffer 2 (proto2) schema definition for an AirQuality message, including optional fields for device name, temperature, and humidity. This file is used to generate Java code for data serialization and deserialization.

```protobuf
syntax = "proto2";

package com.example;

option java_package = "com.example.proto";
option java_outer_classname = "Record";

message AirQuality {
    optional string device_name = 1;
    optional int32 temp = 2;
    optional int64 humidity = 3;
}
```