Hardwood

https://hardwood.dev/latest
Admin
Hardwood is a lightweight, zero-dependency Java library and CLI tool for reading Apache Parquet...
Tokens:25,305
Snippets:286
Trust Score:4.4
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start interactive Docker shell

Source: https://hardwood.dev/latest/reference/cli

Launch an interactive container session with pre-loaded tab completion.

```bash
docker run --rm -it \
  -v "$(pwd)":/data \
  ghcr.io/hardwood-hq/hardwood:1.0.0.Final
```

--------------------------------

### Install libdeflate Native Library

Source: https://hardwood.dev/latest/reference/configuration

System-specific commands to install the libdeflate library required for faster GZIP decompression.

```bash
brew install libdeflate
```

```bash
apt install libdeflate-dev
```

```bash
dnf install libdeflate-devel
```

```bash
vcpkg install libdeflate
```

--------------------------------

### SIMD Status Log Message

Source: https://hardwood.dev/latest/reference/configuration

Example of the INFO log message displayed at startup when SIMD support is successfully enabled.

```text
SIMD support: enabled (256-bit vectors)
```

--------------------------------

### Launch Interactive UI

Source: https://hardwood.dev/latest/reference/cli

Starts the terminal-based interactive explorer for a Parquet file. Requires an interactive terminal session.

```bash
hardwood dive -f data.parquet
```

--------------------------------

### Start JFR Recording

Source: https://hardwood.dev/latest/reference/configuration

Command to initiate a Java Flight Recorder session for performance profiling.

```bash
java -XX:StartFlightRecording=filename=recording.jfr,settings=profile ...
```

--------------------------------

### Accessing advanced nested data structures

Source: https://hardwood.dev/latest/how-to/row-reader

Examples for handling complex nested types including lists of structs, nested lists, and maps.

```java
        // Access list of structs
        PqList contacts = rowReader.getList("contacts");
        if (contacts != null) {
            for (PqStruct contact : contacts.structs()) {
                String contactName = contact.getString("name");
                String phone = contact.getString("phone");
            }
        }

        // Access nested lists (list<list<int>>) using primitive int lists
        PqList matrix = rowReader.getList("matrix");
        if (matrix != null) {
            for (PqList row : matrix.lists()) {
                PqIntList innerList = row.ints();
                for (var it = innerList.iterator(); it.hasNext(); ) {
                    int val = it.nextInt();
                    System.out.println("Value: " + val);
                }
            }
        }

        // Access maps (map<string, int>) — iterate all entries
        PqMap attributes = rowReader.getMap("attributes");
        if (attributes != null) {
            for (PqMap.Entry entry : attributes.getEntries()) {
                String key = entry.getStringKey();
                int value = entry.getIntValue();
                System.out.println(key + " = " + value);
            }
        }

        // Key-based lookup (no per-entry flyweight allocations)
        PqMap attrs = rowReader.getMap("attributes");
        if (attrs != null && attrs.containsKey("age")) {
            Integer age = (Integer) attrs.getValue("age");
        }

        // Access maps with struct values (map<string, struct>)
        PqMap people = rowReader.getMap("people");
        if (people != null) {
            PqStruct alice = (PqStruct) people.getValue("alice");
            if (alice != null) {
                String name = alice.getString("name");
                int age = alice.getInt("age");
            }
        }
```

--------------------------------

### Schema definition for a contacts column

Source: https://hardwood.dev/latest/concepts/nested-columns

Example schema representation of a contacts list containing elements with name and phoneNumber fields.

```text
optional group contacts (LIST) {
  repeated group list {
    optional group element {
      required byte_array name (STRING);
      optional byte_array phoneNumber (STRING);
    }
  }
}
```

--------------------------------

### S3Source Builder Options

Source: https://hardwood.dev/latest/reference/s3

Configuration options for initializing an S3Source instance.

```APIDOC
## S3Source Builder Options

### Description
Configures the S3Source client for connecting to AWS S3 or S3-compatible services.

### Parameters
- **region** (String) - Required (unless endpoint is set) - AWS region.
- **endpoint** (String) - Optional - Custom endpoint for S3-compatible services.
- **pathStyle** (boolean) - Optional - Enables path-style addressing (default: false).
- **credentials** (Object) - Required - Static S3Credentials, S3CredentialsProvider, or AWS chain.
- **connectTimeout** (Duration) - Optional - Max time to establish TCP connection (default: 10s).
- **requestTimeout** (Duration) - Optional - Max time for individual HTTP request (default: 30s).
- **maxRetries** (int) - Optional - GET retries on transient failures (default: 3).
- **httpClient** (HttpClient) - Optional - Caller-supplied client for transport control.
```

--------------------------------

### Download NYC Taxi Data

Source: https://hardwood.dev/latest/tutorial/first-read

Downloads the sample Parquet file required for the tutorial steps.

```bash
curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2026-01.parquet
```

--------------------------------

### Enable Vector API Module

Source: https://hardwood.dev/latest/reference/configuration

JVM argument to enable the incubator module for SIMD acceleration.

```bash
--add-modules jdk.incubator.vector
```

--------------------------------

### Access Parquet Metadata in Java

Source: https://hardwood.dev/latest/how-to/metadata

Demonstrates opening a Parquet file to inspect version, schema, row groups, and column statistics.

```java
import dev.hardwood.metadata.ColumnChunk;
import dev.hardwood.metadata.ColumnMetaData;
import dev.hardwood.metadata.ColumnOrder;
import dev.hardwood.metadata.FileMetaData;
import dev.hardwood.metadata.RowGroup;
import dev.hardwood.metadata.Statistics;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.schema.ColumnSchema;
import dev.hardwood.schema.FileSchema;

import java.util.List;
import java.util.Map;

try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(path))) {
    FileMetaData metadata = reader.getFileMetaData();

    System.out.println("Version: " + metadata.version());
    System.out.println("Total rows: " + metadata.numRows());
    System.out.println("Created by: " + metadata.createdBy());

    // Access application-defined key-value metadata (e.g. Spark schema, pandas metadata, Avro schema)
    Map<String, String> kvMetadata = metadata.keyValueMetadata();
    for (Map.Entry<String, String> entry : kvMetadata.entrySet()) {
        System.out.println("  " + entry.getKey() + " = " + entry.getValue());
    }

    // The statistics ordering for each leaf column, in schema order. Empty when the file omits
    // column_orders, in which case the type-defined ordering applies to every column. The value is
    // one of ColumnOrder.TYPE_DEFINED_ORDER, IEEE754_TOTAL_ORDER, or UNKNOWN.
    List<ColumnOrder> columnOrders = metadata.columnOrders();

    // Schema inspection
    FileSchema schema = reader.getFileSchema();
    for (int i = 0; i < schema.getColumnCount(); i++) {
        ColumnSchema column = schema.getColumn(i);
        System.out.println("Column " + i + ": " + column.name()
            + " (" + column.type() + ", " + column.repetitionType()
            + (column.logicalType() != null ? ", " + column.logicalType() : "")
            + ")");
    }

    // Row group and column chunk details
    for (int rg = 0; rg < metadata.rowGroups().size(); rg++) {
        RowGroup rowGroup = metadata.rowGroups().get(rg);
        System.out.println("Row group " + rg + ": "
            + rowGroup.numRows() + " rows, "
            + rowGroup.totalByteSize() + " bytes");

        for (ColumnChunk chunk : rowGroup.columns()) {
            ColumnMetaData col = chunk.metaData();
            System.out.println("  " + col.pathInSchema()
                + " [" + col.codec() + "]"
                + " compressed=" + col.totalCompressedSize()
                + " uncompressed=" + col.totalUncompressedSize());

            // Column statistics (if available)
            Statistics stats = col.statistics();
            if (stats != null && stats.nullCount() != null) {
                System.out.println("    nulls: " + stats.nullCount());
            }
        }
    }
}
```

--------------------------------

### Define Filter Predicates in Java

Source: https://hardwood.dev/latest/how-to/query-controls

Demonstrates creating simple, compound, IN, and NULL-check filters, and applying them to a ParquetFileReader.

```java
import dev.hardwood.reader.FilterPredicate;

// Simple filter
FilterPredicate filter = FilterPredicate.gt("age", 21);

// Compound filter
FilterPredicate filter = FilterPredicate.and(
    FilterPredicate.gtEq("salary", 50000L),
    FilterPredicate.lt("age", 65)
);

// IN filter
FilterPredicate filter = FilterPredicate.in("department_id", 1, 3, 7);
FilterPredicate filter = FilterPredicate.inStrings("city", "NYC", "LA", "Chicago");

// NULL checks
FilterPredicate filter = FilterPredicate.isNull("middle_name");
FilterPredicate filter = FilterPredicate.isNotNull("email");

try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
     RowReader rowReader = fileReader.buildRowReader().filter(filter).build()) {

    while (rowReader.hasNext()) {
        rowReader.next();
        // Only rows from non-skipped row groups are returned
    }
}
```

--------------------------------

### Configure Hardwood BOM

Source: https://hardwood.dev/latest/getting-started

Import the Hardwood BOM to manage versions for all modules automatically.

```xml
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>dev.hardwood</groupId>
            <artifactId>hardwood-bom</artifactId>
            <version>1.0.0.Final</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
```

```groovy
dependencies {
    implementation platform('dev.hardwood:hardwood-bom:1.0.0.Final')
}
```

--------------------------------

### Read multiple files from S3

Source: https://hardwood.dev/latest/how-to/s3

Demonstrates reading multiple files from a single bucket or across different buckets using the Hardwood library.

```java
import dev.hardwood.Hardwood;

try (Hardwood hardwood = Hardwood.create();
     ParquetFileReader parquet = hardwood.openAll(
             source.inputFilesInBucket("my-bucket",
                     "data/part-001.parquet",
                     "data/part-002.parquet",
                     "data/part-003.parquet"));
     RowReader reader = parquet.rowReader()) {
    while (reader.hasNext()) {
        reader.next();
        // ...
    }
}
```

```java
hardwood.openAll(source.inputFiles(
        "s3://bucket-a/events.parquet",
        "s3://bucket-b/events.parquet"));
```

--------------------------------

### Enable shell completion

Source: https://hardwood.dev/latest/reference/cli

Source the completion script to enable tab completion for commands and arguments.

```bash
source hardwood_completion
```

--------------------------------

### Run Hardwood commands in Docker

Source: https://hardwood.dev/latest/reference/cli

Execute CLI commands by appending them to the docker run command.

```bash
docker run --rm ghcr.io/hardwood-hq/hardwood:1.0.0.Final --help
docker run --rm ghcr.io/hardwood-hq/hardwood:1.0.0.Final info -f /data/data.parquet
```

--------------------------------

### Apply reader options with AvroRowReader

Source: https://hardwood.dev/latest/how-to/avro

Configure column projection and predicate pushdown filters when building an AvroRowReader.

```java
// With filter
AvroRowReader reader = AvroReaders.buildRowReader(fileReader)
    .filter(FilterPredicate.gt("id", 1000L))
    .build();

// With projection
AvroRowReader reader = AvroReaders.buildRowReader(fileReader)
    .projection(ColumnProjection.columns("id", "name"))
    .build();

// With both
AvroRowReader reader = AvroReaders.buildRowReader(fileReader)
    .projection(ColumnProjection.columns("id", "name"))
    .filter(FilterPredicate.gt("id", 1000L))
    .build();
```

--------------------------------

### Configure S3-compatible services

Source: https://hardwood.dev/latest/how-to/s3

Configures S3Source with custom endpoints for services like Cloudflare R2, GCP, and MinIO. Use pathStyle(true) for services requiring path-style access.

```java
// Cloudflare R2
S3Source source = S3Source.builder()
        .endpoint("https://<account-id>.r2.cloudflarestorage.com")
        .credentials(S3Credentials.of(accessKeyId, secretKey))
        .build();

// GCP Cloud Storage (HMAC keys)
S3Source source = S3Source.builder()
        .endpoint("https://storage.googleapis.com")
        .credentials(S3Credentials.of(hmacAccessId, hmacSecret))
        .build();

// MinIO (path-style)
S3Source source = S3Source.builder()
        .endpoint("http://localhost:9000")
        .pathStyle(true)
        .credentials(S3Credentials.of(accessKeyId, secretKey))
        .build();
```

--------------------------------

### Add Optional Compression Dependencies via BOM

Source: https://hardwood.dev/latest/getting-started

Include optional compression libraries without specifying versions when using the BOM.

```xml
<dependency>
    <groupId>org.xerial.snappy</groupId>
    <artifactId>snappy-java</artifactId>
</dependency>
```

```groovy
implementation 'org.xerial.snappy:snappy-java'
```

--------------------------------

### Run TUI in Docker

Source: https://hardwood.dev/latest/reference/cli

Pass the -it flag to provide an interactive terminal for TUI commands.

```bash
docker run --rm -it \
  -v "$(pwd)":/data \
  ghcr.io/hardwood-hq/hardwood:1.0.0.Final \
  dive -f /data/data.parquet
```

--------------------------------

### Add Optional Compression Dependencies Explicitly

Source: https://hardwood.dev/latest/getting-started

Include optional compression libraries with explicit versions when not using the BOM.

```xml
<dependency>
    <groupId>org.xerial.snappy</groupId>
    <artifactId>snappy-java</artifactId>
    <version>1.1.10.8</version>
</dependency>
<dependency>
    <groupId>com.github.luben</groupId>
    <artifactId>zstd-jni</artifactId>
    <version>1.5.7-6</version>
</dependency>
<dependency>
    <groupId>at.yawk.lz4</groupId>
    <artifactId>lz4-java</artifactId>
    <version>1.8.1</version>
</dependency>
<dependency>
    <groupId>com.aayushatharva.brotli4j</groupId>
    <artifactId>brotli4j</artifactId>
    <version>1.20.0</version>
</dependency>
```

```groovy
implementation 'org.xerial.snappy:snappy-java:1.1.10.8'
implementation 'com.github.luben:zstd-jni:1.5.7-6'
implementation 'at.yawk.lz4:lz4-java:1.8.1'
implementation 'com.aayushatharva.brotli4j:brotli4j:1.20.0'
```

--------------------------------

### Enable Native Access for libdeflate

Source: https://hardwood.dev/latest/reference/configuration

Required JVM flag to allow Hardwood to bind native functions for GZIP decompression.

```bash
--enable-native-access=ALL-UNNAMED
```

--------------------------------

### Read multiple Parquet files

Source: https://hardwood.dev/latest/how-to/multi-file

Use Hardwood.openAll to process a list of InputFile objects as a single dataset.

```java
import dev.hardwood.Hardwood;
import dev.hardwood.InputFile;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

List<InputFile> files = InputFile.ofPaths(
    Path.of("data_2024_01.parquet"),
    Path.of("data_2024_02.parquet"),
    Path.of("data_2024_03.parquet")
);

try (Hardwood hardwood = Hardwood.create();
     ParquetFileReader parquet = hardwood.openAll(files);
     RowReader reader = parquet.rowReader()) {

    while (reader.hasNext()) {
        reader.next();
        // Access data using the same API as a single-file RowReader
        long id = reader.getLong("id");
        String name = reader.getString("name");
    }
}
```

--------------------------------

### Apply filter pushdown

Source: https://hardwood.dev/latest/how-to/compat

Use standard FilterApi and FilterCompat classes to push predicates down to the reader for improved performance.

```java
import static org.apache.parquet.filter2.predicate.FilterApi.*;
import org.apache.parquet.filter2.compat.FilterCompat;

FilterPredicate pred = and(
    gtEq(longColumn("id"), 100L),
    lt(doubleColumn("amount"), 500.0)
);

try (ParquetReader<Group> reader = ParquetReader.builder(new GroupReadSupport(), path)
        .withFilter(FilterCompat.get(pred))
        .build()) {
    // only rows from matching row groups are returned
}
```

--------------------------------

### Read Map Columns in Lockstep

Source: https://hardwood.dev/latest/how-to/column-reader

Demonstrates how to open key and value columns and iterate through them using shared layer offsets.

```java
try (ColumnReader keys   = reader.columnReader("tags.key_value.key");
     ColumnReader values = reader.columnReader("tags.key_value.value")) {

    while (keys.nextBatch() & values.nextBatch()) {
        int recordCount        = keys.getRecordCount();
        int[]    entryOffsets  = keys.getLayerOffsets(0);
        Validity mapValidity   = keys.getLayerValidity(0);
        byte[]   keyBytes      = keys.getBinaryValues();
        int[]    keyOffsets    = keys.getBinaryOffsets();
        int[]    valueInts     = values.getInts();
        Validity valueValidity = values.getLeafValidity();
        boolean  anyNullMap   = mapValidity.hasNulls();
        boolean  anyNullValue = valueValidity.hasNulls();

        for (int r = 0; r < recordCount; r++) {
            if (anyNullMap && mapValidity.isNull(r)) continue;  // null map
            int start = entryOffsets[r];
            int end   = entryOffsets[r + 1];
            for (int i = start; i < end; i++) {
                int keyStart = keyOffsets[i];
                int keyLen   = keyOffsets[i + 1] - keyStart;
                String key   = new String(keyBytes, keyStart, keyLen, StandardCharsets.UTF_8);

                if (anyNullValue && valueValidity.isNull(i)) {
                    // key present, value null
                } else {
                    process(key, valueInts[i]);
                }
            }
        }
    }
}
```

--------------------------------

### Mount local files in Docker

Source: https://hardwood.dev/latest/reference/cli

Use the -v flag to mount a local directory into the container for file access.

```bash
docker run --rm \
  -v "$(pwd)":/data \
  ghcr.io/hardwood-hq/hardwood:1.0.0.Final \
  schema -f /data/data.parquet
```

--------------------------------

### Read S3 file with static credentials

Source: https://hardwood.dev/latest/how-to/s3

Configure an S3Source with static credentials and read a Parquet file using ParquetFileReader.

```java
import dev.hardwood.s3.S3Credentials;
import dev.hardwood.s3.S3Source;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(S3Credentials.of("AKIA...", "secret"))
        .build();

try (ParquetFileReader reader = ParquetFileReader.open(
        source.inputFile("s3://my-bucket/data/trips.parquet"))) {
    try (RowReader rows = reader.rowReader()) {
        while (rows.hasNext()) {
            rows.next();
            long id = rows.getLong("id");
        }
    }
}
```

--------------------------------

### Specify Hardwood Core Version Directly

Source: https://hardwood.dev/latest/getting-started

Add the core library dependency without using the BOM.

```xml
<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-core</artifactId>
    <version>1.0.0.Final</version>
</dependency>
```

```groovy
implementation 'dev.hardwood:hardwood-core:1.0.0.Final'
```

--------------------------------

### Handle NaN and Signed Zeros in Comparisons

Source: https://hardwood.dev/latest/how-to/query-controls

Illustrates matching NaN values and both signed zeros using total order comparison semantics.

```java
// Match any NaN row
FilterPredicate anyNaN = FilterPredicate.eq("score", Double.NaN);

// Match both signed zeros
FilterPredicate anyZero = FilterPredicate.or(
    FilterPredicate.eq("ratio", 0.0),
    FilterPredicate.eq("ratio", -0.0)
);
```

--------------------------------

### Read S3 files with CLI

Source: https://hardwood.dev/latest/reference/cli

Use the -f flag with an s3:// URI to process files directly from S3 buckets.

```bash
hardwood schema -f s3://my-bucket/data.parquet
hardwood print -n 10 -f s3://my-bucket/data.parquet
```

--------------------------------

### Configure decode parallelism

Source: https://hardwood.dev/latest/how-to/multi-file

Control the thread pool size by creating and passing a HardwoodContext to Hardwood.create.

```java
import dev.hardwood.HardwoodContext;

try (HardwoodContext context = HardwoodContext.create(4);  // 4 threads
     Hardwood hardwood = Hardwood.create(context);
     ParquetFileReader parquet = hardwood.openAll(files);
     RowReader reader = parquet.rowReader()) {
    // ...
}
```

--------------------------------

### Fetch-cost counters

Source: https://hardwood.dev/latest/reference/s3

Methods to track network I/O performance for S3InputFile.

```APIDOC
## S3InputFile Fetch-cost counters

### Description
Methods to retrieve network I/O statistics performed since the file was opened.

### Methods
- **networkRequestCount()** - Returns the number of HTTP GET requests issued.
- **networkBytesFetched()** - Returns the total bytes fetched across network requests.
```

--------------------------------

### Pull Hardwood Docker image

Source: https://hardwood.dev/latest/reference/cli

Download the official Fedora-based Docker image from the GitHub Container Registry.

```bash
docker pull ghcr.io/hardwood-hq/hardwood:1.0.0.Final
```

--------------------------------

### Hardwood CLI Batch Commands

Source: https://hardwood.dev/latest/reference/cli

Common batch operations for inspecting and converting Parquet files.

```bash
# Show file overview
hardwood info -f data.parquet

# Print schema
hardwood schema -f data.parquet

# Show first 20 rows
hardwood print -n 20 -f data.parquet

# Show last 5 rows
hardwood print -n -5 -f data.parquet

# Show all rows
hardwood print -f data.parquet

# Convert to CSV
hardwood convert --format csv -f data.parquet

# Show dictionary entries for a column (first 50 entries per row group by default)
hardwood inspect dictionary -f data.parquet -c category

# Show all dictionary entries for a column (--limit 0 means unlimited)
hardwood inspect dictionary -f data.parquet -c category --limit 0

# Convert first 100 rows to JSON
hardwood convert -n 100 --format json -f data.parquet

# Convert last 50 rows to CSV
hardwood convert -n -50 --format csv -f data.parquet
```

--------------------------------

### Declare Hardwood Core Dependency

Source: https://hardwood.dev/latest/getting-started

Add the core library dependency to your project after configuring the BOM.

```xml
<dependencies>
    <dependency>
        <groupId>dev.hardwood</groupId>
        <artifactId>hardwood-core</artifactId>
    </dependency>
</dependencies>
```

--------------------------------

### Add hardwood-avro dependency

Source: https://hardwood.dev/latest/how-to/avro

Include the hardwood-avro module in your project to enable Avro record materialization.

```xml
<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-avro</artifactId>
</dependency>
```

--------------------------------

### Combining Column Projection and Predicate Pushdown

Source: https://hardwood.dev/latest/how-to/query-controls

Apply both column projection and filter predicates simultaneously to optimize data retrieval. Filter columns do not need to be included in the projection to be used for pushdown.

```java
try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
     RowReader rowReader = fileReader.buildRowReader()
             .projection(ColumnProjection.columns("id", "name", "salary"))
             .filter(FilterPredicate.gtEq("salary", 50000L))
             .build()) {

    while (rowReader.hasNext()) {
        rowReader.next();
        long id = rowReader.getLong("id");
        String name = rowReader.getString("name");
        long salary = rowReader.getLong("salary");
    }
}
```

--------------------------------

### Configure S3Source Timeouts

Source: https://hardwood.dev/latest/how-to/s3

Sets custom connect and request timeouts for the S3 source builder.

```java
S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(S3Credentials.of("AKIA...", "secret"))
        .connectTimeout(Duration.ofSeconds(5))   // default: 10s
        .requestTimeout(Duration.ofSeconds(60))   // default: 30s
        .build();
```

--------------------------------

### Add hardwood-aws-auth dependency

Source: https://hardwood.dev/latest/how-to/s3

Include the optional hardwood-aws-auth module to support the standard AWS credential chain.

```xml
<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-aws-auth</artifactId>
</dependency>
```

--------------------------------

### Reading Parquet files with RowReader

Source: https://hardwood.dev/latest/how-to/row-reader

Demonstrates opening a Parquet file and iterating through rows to access fields using typed accessor methods.

```java
import dev.hardwood.InputFile;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;
import dev.hardwood.row.PqStruct;
import dev.hardwood.row.PqList;
import dev.hardwood.row.PqIntList;
import dev.hardwood.row.PqMap;
import java.time.Instant;
import java.time.LocalDate;
import java.time.LocalTime;
import java.math.BigDecimal;
import java.util.UUID;

try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
    RowReader rowReader = fileReader.rowReader()) {

    while (rowReader.hasNext()) {
        rowReader.next();

        // Access columns by name with typed accessors
        long id = rowReader.getLong("id");
        String name = rowReader.getString("name");

        // Logical types are automatically converted
        LocalDate birthDate = rowReader.getDate("birth_date");
        Instant createdAt = rowReader.getTimestamp("created_at");
        LocalTime wakeTime = rowReader.getTime("wake_time");
        BigDecimal balance = rowReader.getDecimal("balance");
        UUID accountId = rowReader.getUuid("account_id");

        // Check for null values
        if (!rowReader.isNull("age")) {
            int age = rowReader.getInt("age");
            System.out.println("ID: " + id + ", Name: " + name + ", Age: " + age);
        }

        // Access nested structs
        PqStruct address = rowReader.getStruct("address");
        if (address != null) {
            String city = address.getString("city");
            int zip = address.getInt("zip");
        }

        // Access lists and iterate with typed accessors
        PqList tags = rowReader.getList("tags");
        if (tags != null) {
            for (String tag : tags.strings()) {
                System.out.println("Tag: " + tag);
            }
        }
    }
}
```

--------------------------------

### Replicating parquet-java notEq semantics

Source: https://hardwood.dev/latest/concepts/compatibility-philosophy

Explicitly combine a not-equal filter with an is-null check to match parquet-java's behavior of including null rows.

```java
// parquet-java's notEq semantics: "not equal to v, OR null"
FilterPredicate.or(FilterPredicate.notEq("x", v), FilterPredicate.isNull("x"));
```

--------------------------------

### Add hardwood-s3 dependency

Source: https://hardwood.dev/latest/how-to/s3

Include the hardwood-s3 module in your project to enable S3 integration.

```xml
<dependency>
    <groupId>dev.hardwood</groupId>
    <artifactId>hardwood-s3</artifactId>
</dependency>
```

--------------------------------

### Use dynamic credentials provider

Source: https://hardwood.dev/latest/how-to/s3

Implement the S3CredentialsProvider interface to support refreshable or dynamic credentials.

```java
S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(() -> fetchCredentialsFromVault())
        .build();
```

--------------------------------

### Read local Parquet files with ParquetReader

Source: https://hardwood.dev/latest/how-to/compat

Use the standard ParquetReader API to read local files. Ensure the hardwood-parquet-java-compat module is on the classpath instead of the original parquet-java library.

```java
import org.apache.hadoop.fs.Path;
import org.apache.parquet.example.data.Group;
import org.apache.parquet.hadoop.GroupReadSupport;
import org.apache.parquet.hadoop.ParquetReader;

Path path = new Path("data.parquet");

try (ParquetReader<Group> reader = ParquetReader.builder(new GroupReadSupport(), path).build()) {
    Group record;
    while ((record = reader.read()) != null) {
        long id = record.getLong("id", 0);
        String name = record.getString("name", 0);

        Group address = record.getGroup("address", 0);
        String city = address.getString("city", 0);
    }
}
```

--------------------------------

### Add JTS Core Dependency

Source: https://hardwood.dev/latest/how-to/geospatial

Include the JTS Topology Suite dependency in your project to enable WKB decoding.

```xml
<dependency>
    <groupId>org.locationtech.jts</groupId>
    <artifactId>jts-core</artifactId>
    <version>1.20.0</version>
</dependency>
```

--------------------------------

### Read a Parquet file with Hardwood

Source: https://hardwood.dev/latest

Use ParquetFileReader and RowReader to iterate through rows and access column values by name.

```java
import dev.hardwood.InputFile;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
    RowReader rowReader = fileReader.rowReader()) {

    while (rowReader.hasNext()) {
        rowReader.next();

        long id = rowReader.getLong("id");
        String name = rowReader.getString("name");
        LocalDate birthDate = rowReader.getDate("birth_date");
        Instant createdAt = rowReader.getTimestamp("created_at");
    }
}
```

--------------------------------

### Disable libdeflate

Source: https://hardwood.dev/latest/reference/configuration

System property to force the use of the built-in Java GZIP implementation.

```bash
-Dhardwood.uselibdeflate=false
```

--------------------------------

### Hand Off Batch Arrays to Threads

Source: https://hardwood.dev/latest/how-to/column-reader

Demonstrates that batch arrays are safe to process in separate threads after being retrieved from the reader.

```java
while (columns.nextBatch()) {
    int count = columns.getRecordCount();
    long[]   v0 = columns.getColumnReader("passenger_count").getLongs();
    double[] v1 = columns.getColumnReader("trip_distance").getDoubles();
    double[] v2 = columns.getColumnReader("fare_amount").getDoubles();
    Thread.ofPlatform().start(() -> process(count, v0, v1, v2));
}
```

--------------------------------

### Hoisting null checks for analytical workloads

Source: https://hardwood.dev/latest/how-to/column-reader

Use this pattern to optimize performance by checking for nulls once per batch rather than per element.

```java
Validity validity = col.getLeafValidity();
boolean hasNulls = validity.hasNulls();
for (int i = 0; i < count; i++) {
    if (!hasNulls || validity.isNotNull(i)) {
        sum += values[i];
    }
}
```

--------------------------------

### Projecting Columns in ParquetFileReader

Source: https://hardwood.dev/latest/how-to/query-controls

Use ColumnProjection to specify a subset of columns to read, which improves performance by reducing I/O and memory usage. Accessing columns not included in the projection will result in an IllegalArgumentException.

```java
import dev.hardwood.InputFile;
import dev.hardwood.schema.ColumnProjection;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
     RowReader rowReader = fileReader.buildRowReader()
             .projection(ColumnProjection.columns("id", "name", "created_at"))
             .build()) {

    while (rowReader.hasNext()) {
        rowReader.next();

        // Access projected columns normally
        long id = rowReader.getLong("id");
        String name = rowReader.getString("name");
        Instant createdAt = rowReader.getTimestamp("created_at");

        // Accessing non-projected columns throws IllegalArgumentException
        // rowReader.getInt("age");  // throws "Column not in projection: age"
    }
}
```

--------------------------------

### Use AWS default credential chain

Source: https://hardwood.dev/latest/how-to/s3

Configure S3Source to use the default AWS credential chain via the hardwood-aws-auth module.

```java
import dev.hardwood.aws.auth.SdkCredentialsProviders;

S3Source source = S3Source.builder()
        .region("us-east-1")
        .credentials(SdkCredentialsProviders.defaultChain())
        .build();
```

--------------------------------

### Read Parquet files from S3

Source: https://hardwood.dev/latest/how-to/compat

Configure Hadoop-style S3 credentials and endpoints to read files from S3. Requires the hardwood-s3 dependency to be present on the classpath.

```java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.parquet.hadoop.util.HadoopInputFile;

Configuration conf = new Configuration();
conf.set("fs.s3a.access.key", "...");
conf.set("fs.s3a.secret.key", "...");
conf.set("fs.s3a.endpoint", "https://s3.us-east-1.amazonaws.com");

Path path = new Path("s3a://my-bucket/data.parquet");

try (ParquetReader<Group> reader = ParquetReader.builder(new GroupReadSupport(), path)
        .withConf(conf)
        .build()) {
    // read as usual
}
```

--------------------------------

### Configure Batch Size

Source: https://hardwood.dev/latest/how-to/column-reader

Sets a specific record count per batch using the batchSize builder method.

```java
try (ColumnReaders columns = parquet.buildColumnReaders(
             ColumnProjection.columns("passenger_count", "trip_distance", "fare_amount"))
        .batchSize(2048)
        .build()) {
    // ...
}
```

--------------------------------

### Word-wise iteration for null-dense regions

Source: https://hardwood.dev/latest/how-to/column-reader

Iterate over backing long arrays to skip nulls efficiently in null-dense columns using bit manipulation.

```java
Validity validity = col.getLeafValidity();
long[] words = validity.words();
if (words == null) {
    // Validity.NO_NULLS — tight loop over every position.
    for (int i = 0; i < count; i++) sum += values[i];
} else {
    int wordCount = (count + 63) >>> 6;
    for (int w = 0; w < wordCount; w++) {
        long present = words[w];
        while (present != 0L) {
            int bit = Long.numberOfTrailingZeros(present);
            sum += values[(w << 6) + bit];
            present &= present - 1L;
        }
    }
}
```
Tokens:
Raw