### Asynchronous Parquet Write Example Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Provides an example of writing Parquet files asynchronously using `tokio`. This requires the 'async' feature to be enabled. The pattern is similar to synchronous writing but uses `async`/`await` for file operations and the `FileStreamer`. ```rust #[tokio::main] async fn async_write_example() -> parquet2::error::Result<()> { use parquet2::write::FileStreamer; let file = tokio::fs::File::create("output.parquet").await?; let mut writer = FileStreamer::new(file); // Async write operations writer.finish().await?; Ok(()) } ``` -------------------------------- ### Use CompressedDataPage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Example of creating and accessing information from a CompressedDataPage. Ensure you have the necessary imports for `CompressedDataPage` and `Compression`. ```rust use parquet2::page::CompressedDataPage; use parquet2::compression::Compression; let page = CompressedDataPage::new( header, compressed_data, Compression::Gzip, 1024, descriptor, Some(100), // 100 rows ); println!("Compressed: {}", page.compressed_size()); println!("Uncompressed: {}", page.uncompressed_size()); ``` -------------------------------- ### ColumnOrder Usage Example Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Demonstrates how to match on the ColumnOrder enum to determine the ordering strategy for a column. ```rust match metadata.column_order(0) { ColumnOrder::TypeDefinedOrder(sort_order) => { println!("Column uses standard ordering"); } ColumnOrder::Undefined => { println!("Column order is undefined"); } } ``` -------------------------------- ### Example: Comparing Binary Strings Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Demonstrates using `ord_binary` to compare two byte slices, showing their lexicographical order. ```rust let a = b"hello"; let b = b"world"; assert_eq!(ord_binary(a, b), std::cmp::Ordering::Less); ``` -------------------------------- ### Handle Column Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Example demonstrating how to safely unwrap statistics and handle different physical types, specifically for Int32. ```rust if let Some(result) = column_meta.statistics() { let stats = result?; match stats.physical_type() { PhysicalType::Int32 => { let int_stats = stats .as_any() .downcast_ref::>() .unwrap(); println!("Min: {:?}", int_stats.min_value); println!("Max: {:?}", int_stats.max_value); } _ => {} } } ``` -------------------------------- ### Get Column Descriptor and Physical Type Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Example showing how to access the column descriptor and its physical type. ```rust let descriptor = column_meta.descriptor(); let physical_type = descriptor.descriptor.primitive_type.physical_type; ``` -------------------------------- ### Initialize FileWriter Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Demonstrates the initial setup for writing a Parquet file. This involves creating a file handle and initializing the FileWriter with basic options. ```rust use parquet2::write::{FileWriter, WriteOptions, Version, RowGroupIter}; use parquet2::compression::CompressionOptions; use parquet2::page::CompressedPage; // Create writer let file = std::fs::File::create("output.parquet")?; let mut writer = FileWriter::new(file); // Configure options let options = WriteOptions { write_statistics: true, version: Version::V2, }; // Write metadata and row groups // (specific API depends on data source) ``` -------------------------------- ### Example: Using NativeType for i32 Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Demonstrates converting an i32 value to little-endian bytes and back using the `NativeType` trait methods. ```rust use parquet2::types::NativeType; let value: i32 = 42; let bytes = value.to_le_bytes(); let restored = i32::from_le_bytes(bytes); assert_eq!(value, restored); ``` -------------------------------- ### Run Integration Tests with PyArrow Source: https://github.com/jorgecarleitao/parquet2/blob/main/README.md Set up a virtual environment, install pyarrow, and generate test files before running cargo tests. This ensures compatibility with pyarrow implementations. ```bash python3 -m venv venv virtualenv/bin/pip install pip --upgrade virtualenv/bin/pip install pyarrow==7 virtualenv/bin/python tests/write_pyarrow.py cargo test ``` -------------------------------- ### Match Column Physical Type Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Example demonstrating how to use a match statement to handle different physical types of a column. ```rust match column_meta.physical_type() { PhysicalType::Int32 => { /* 32-bit integers */ } PhysicalType::ByteArray => { /* strings */ } _ => { /* other types */ } } ``` -------------------------------- ### Iterate Row Groups and Columns Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Example demonstrating how to iterate through row groups and their respective columns, printing column descriptors. ```rust for row_group in &metadata.row_groups { let num_rows = row_group.num_rows(); let columns = row_group.columns(); for column in columns { println!("Column: {}", column.descriptor()); } } ``` -------------------------------- ### Example: Decoding an i32 from Bytes Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Shows how to use the `decode` function to convert a byte slice representing an i32 into its native Rust type. ```rust let bytes: &[u8] = &[42, 0, 0, 0]; // little-endian i32 let value: i32 = decode(bytes); assert_eq!(value, 42); ``` -------------------------------- ### Example: Using Statistics for Page Filtering Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Demonstrates how to read Parquet metadata and use column statistics to determine if a column chunk can be skipped during data processing. ```APIDOC ## Example: Using Statistics for Page Filtering ```rust use parquet2::read::read_metadata; use parquet2::statistics::PrimitiveStatistics; use parquet2::schema::types::PhysicalType; fn filter_pages(path: &str, min_value: i32, max_value: i32) -> Result<()> { let mut file = std::fs::File::open(path)?; let metadata = read_metadata(&mut file)?; for row_group in &metadata.row_groups { for column in row_group.columns() { if column.physical_type() != PhysicalType::Int32 { continue; } if let Some(result) = column.statistics() { let stats = result?; let int_stats = stats .as_any() .downcast_ref::>() .unwrap(); // Check if this column chunk can be skipped if let (Some(min), Some(max)) = (int_stats.min_value, int_stats.max_value) { if max < min_value || min > max_value { println!("Column chunk can be skipped"); continue; } } println!("Column chunk might contain relevant data"); } } } Ok(()) } ``` ``` -------------------------------- ### Creating a Parquet Writer Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Initialize a new Parquet file writer with `write::FileWriter::new()`. This is the starting point for writing data to a Parquet file. ```rust write::FileWriter::new() ``` -------------------------------- ### Get Compression Algorithm Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the compression algorithm used for this column chunk. ```rust pub fn compression(&self) -> Compression ``` -------------------------------- ### Enable Compression Algorithms with Cargo Features Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/API_REFERENCE.md Specify desired compression algorithms and features in your Cargo.toml file to enable them for the parquet2 dependency. This example shows enabling snappy, gzip, lz4, zstd, brotli, and async support. ```toml [dependencies] parquet2 = { version = "0.17", features = ["snappy", "gzip", "lz4", "zstd", "brotli", "async"] } ``` -------------------------------- ### Read a Parquet File Entry Points Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Sequence of operations to read a Parquet file, starting with metadata retrieval and page iteration. ```rust read::read_metadata() ``` ```rust read::get_page_iterator() ``` ```rust read::decompress() ``` -------------------------------- ### Split DataPage Buffer Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Example of splitting the buffer of a DataPage into repetition levels, definition levels, and values. This requires the `split_buffer` function. ```rust use parquet2::page::split_buffer; let (rep_levels, def_levels, values) = split_buffer(&data_page)?; ``` -------------------------------- ### Filter Pages Using Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Example demonstrating how to use column statistics to determine if a column chunk can be skipped during read operations, optimizing data retrieval. This function specifically handles `Int32` types. ```rust use parquet2::read::read_metadata; use parquet2::statistics::PrimitiveStatistics; use parquet2::schema::types::PhysicalType; fn filter_pages(path: &str, min_value: i32, max_value: i32) -> Result<()> { let mut file = std::fs::File::open(path)?; let metadata = read_metadata(&mut file)?; for row_group in &metadata.row_groups { for column in row_group.columns() { if column.physical_type() != PhysicalType::Int32 { continue; } if let Some(result) = column.statistics() { let stats = result?; let int_stats = stats .as_any() .downcast_ref::>() .unwrap(); // Check if this column chunk can be skipped if let (Some(min), Some(max)) = (int_stats.min_value, int_stats.max_value) { if max < min_value || min > max_value { println!("Column chunk can be skipped"); continue; } } println!("Column chunk might contain relevant data"); } } } Ok(()) } ``` -------------------------------- ### Get Data Page Offset Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the byte offset within the file where the first data page for this column chunk is located. ```rust pub fn data_page_offset(&self) -> i64 ``` -------------------------------- ### Retrieving Parquet Page Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Get the minimum and maximum values for pages in a column with `read::read_columns_indexes()`. This is helpful for data filtering and analysis. ```rust read::read_columns_indexes() ``` -------------------------------- ### Access Primitive Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Retrieve statistics for a column chunk, including min/max values. This example specifically handles `Int32` types. ```rust use parquet2::statistics::PrimitiveStatistics; use parquet2::schema::types::PhysicalType; if let Some(result) = column_meta.statistics() { let stats = result?; if stats.physical_type() == &PhysicalType::Int32 { let int_stats = stats .as_any() .downcast_ref::>() .unwrap(); println!("Min: {:?}", int_stats.min_value); println!("Max: {:?}", int_stats.max_value); } } ``` -------------------------------- ### WouldOverAllocate Error Example Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Demonstrates a scenario where the WouldOverAllocate error might be returned. This occurs when an operation is expected to exceed memory limits, such as a page being larger than the maximum allowed size. ```rust let max_page_size = 256 * 1024 * 1024; // 256 MB limit let pages = get_page_iterator( column_meta, &mut file, None, vec![], max_page_size )?; // If page exceeds max_page_size: // Err(Error::WouldOverAllocate) ``` -------------------------------- ### Handling OutOfSpec Errors in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Example of how to match and handle the `OutOfSpec` error when reading Parquet metadata. This is useful for validating file integrity and diagnosing format issues. ```rust match read_metadata(&mut file) { Ok(metadata) => { /* process */ } Err(Error::OutOfSpec(msg)) => { eprintln!("Invalid Parquet file: {}", msg); } Err(e) => eprintln!("Other error: {}", e), } ``` -------------------------------- ### Handle Parquet2 Errors in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Example of handling `Result` from Parquet2 operations, specifically catching and reporting a `FeatureNotActive` error for the Snappy compression codec. ```rust match read_metadata(&mut file) { Ok(metadata) => { /* use metadata */ }, Err(Error::FeatureNotActive(Feature::Snappy, _)) => { eprintln!("Enable snappy feature"); }, Err(e) => eprintln!("Error: {}", e), } ``` -------------------------------- ### Delta Bitpacked Decoder Usage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/encoding_and_bloom_filter.md Example of using the Delta Bitpacked Decoder to read delta-encoded integers from a buffer. Ensure the buffer contains valid delta-encoded data. ```rust use parquet2::encoding::delta_bitpacked::Decoder; let mut decoder = Decoder::new(buffer); while let Some(value) = decoder.next()? { // Process value } ``` -------------------------------- ### Using CompressionOptions in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Demonstrates how to instantiate various `CompressionOptions` in Rust. This includes setting up uncompressed, default level gzip, and custom level Zstd and Brotli compression. ```rust use parquet2::compression::CompressionOptions; // No compression let opt = CompressionOptions::Uncompressed; // Gzip with default level let opt = CompressionOptions::Gzip(None); // Gzip with specific level let opt = CompressionOptions::Gzip(Some(GzipLevel::default())); // Zstd with custom level let opt = CompressionOptions::Zstd(Some(ZstdLevel::new(19))); // Brotli with custom level let opt = CompressionOptions::Brotli(Some(BrotliLevel::new(11))); ``` -------------------------------- ### Get Root ParquetType from SchemaDescriptor Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the root `ParquetType` of the schema. This provides access to the entire schema hierarchy starting from the top. ```rust pub fn root(&self) -> &ParquetType ``` -------------------------------- ### Getting Parquet File Information in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Reads and prints metadata from a Parquet file, including version, total rows, creator, and row group details. Requires a valid file path and appropriate file system permissions. ```rust use parquet2::read::read_metadata; fn file_info(path: &str) -> parquet2::error::Result<()> { let mut file = std::fs::File::open(path)?; let metadata = read_metadata(&mut file)?; println!("Parquet Version: {}", metadata.version); println!("Total Rows: {}", metadata.num_rows); println!("Created By: {}", metadata.created_by.as_deref().unwrap_or("unknown")); println!("Row Groups: {}", metadata.row_groups.len()); for (i, rg) in metadata.row_groups.iter().enumerate() { println!(" RG {}: {} rows, {} columns", i, rg.num_rows(), rg.columns().len()); } Ok(()) } ``` -------------------------------- ### Handling FeatureNotActive Error Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Example of matching and handling the FeatureNotActive error, specifically for Snappy compression. ```rust use parquet2::error::{Error, Feature}; match decompress(Compression::Snappy, input, &mut output) { Ok(_) => { /* decompressed */ } Err(Error::FeatureNotActive(Feature::Snappy, reason)) => { eprintln!("Enable snappy feature to {}", reason); } Err(e) => eprintln!("Other error: {}", e), } ``` -------------------------------- ### Parquet Tools Usage Source: https://github.com/jorgecarleitao/parquet2/blob/main/parquet-tools/README.md Basic usage instructions for the parquet-tools CLI. Specify the Parquet file and the desired subcommand. ```bash USAGE: parquet-tools [SUBCOMMAND] FLAGS: -h, --help Prints help information -V, --version Prints version information ARGS: Parquet file to read SUBCOMMANDS: dump dumps data from column help Prints this message or the help of the given subcommand(s) meta meta information about the file rowcount number of rows from file ``` -------------------------------- ### ParquetType::name Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/schema_types.md Gets the name of the field represented by the ParquetType. This is applicable to both primitive and group types. ```APIDOC ## ParquetType::name ### Description Returns the field name. ### Method ```rust pub fn name(&self) -> &str ``` ``` -------------------------------- ### Write a Parquet File Entry Points Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Steps involved in writing a Parquet file, from initialization to finalization. ```rust write::FileWriter::new() ``` ```rust write::WriteOptions ``` ```rust FileWriter::finish() ``` -------------------------------- ### Get Uncompressed Size of Column Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the total uncompressed size of the column data in bytes. ```rust pub fn uncompressed_size(&self) -> i64 ``` -------------------------------- ### Get Compressed Size of Column Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the total compressed size of the column data in bytes. ```rust pub fn compressed_size(&self) -> i64 ``` -------------------------------- ### Initialize Parquet FileWriter Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/API_REFERENCE.md Instantiate `FileWriter` to begin writing a Parquet file. The `output` parameter should be a suitable writer. Subsequent operations involve writing row groups and pages. ```rust use parquet2::write::FileWriter; let writer = FileWriter::new(output); // write row groups and pages ``` -------------------------------- ### Handling FeatureNotSupported Error Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Example of checking for and returning a FeatureNotSupported error for Byte Stream Split encoding. ```rust match page.encoding() { Encoding::ByteStreamSplit => { return Err(Error::FeatureNotSupported( "Byte Stream Split encoding not yet supported".into() )); } _ => {} } ``` -------------------------------- ### Basic Synchronous Parquet Write Structure Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Demonstrates the fundamental structure for writing a Parquet file synchronously. It involves creating a file, initializing a `FileWriter` with options, and finishing the write operation. Ensure necessary imports are included. ```rust use parquet2::write::{FileWriter, WriteOptions, Version}; use parquet2::metadata::FileMetaData; use std::fs::File; fn write_parquet( path: &str, metadata: FileMetaData, ) -> parquet2::error::Result<()> { let file = File::create(path)?; let mut writer = FileWriter::new(file); let options = WriteOptions { write_statistics: true, version: Version::V2, }; // Write row groups and metadata using writer interface // (Specific method depends on data format) writer.finish()?; Ok(()) } ``` -------------------------------- ### Get Name for ParquetType Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/schema_types.md Returns the name of the field represented by the ParquetType. Applicable to both primitive and group types. ```rust pub fn name(&self) -> &str ``` -------------------------------- ### Get Column Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves deserialized statistics for the column, if available. Returns an Option containing a Result. ```rust pub fn statistics(&self) -> Option>> ``` -------------------------------- ### Get Physical Type of Column Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the physical data type of the column (e.g., Int32, ByteArray). ```rust pub fn physical_type(&self) -> PhysicalType ``` -------------------------------- ### Get Field Info for ParquetType Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/schema_types.md Retrieves the FieldInfo associated with a ParquetType. Useful for accessing metadata about a schema element. ```rust pub fn get_field_info(&self) -> &FieldInfo ``` -------------------------------- ### Get ColumnChunkMetaData for RowGroup Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns a slice of all column chunks present in this row group. Useful for iterating through columns. ```rust pub fn columns(&self) -> &[ColumnChunkMetaData] ``` -------------------------------- ### Create DataPage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Constructs a new uncompressed data page. Use this for data that is not compressed. ```rust pub fn new( header: DataPageHeader, buffer: Vec, descriptor: Descriptor, rows: Option, ) -> Self ``` -------------------------------- ### Get Number of Rows in RowGroupMetaData Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the total number of rows contained within a specific row group. ```rust pub fn num_rows(&self) -> i64 ``` -------------------------------- ### Enabling Features for Compression Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Shows how to enable specific compression features in Cargo.toml. ```toml [dependencies] parquet2 = { version = "0.17", features = ["snappy", "gzip", "lz4", "zstd", "brotli"] } ``` ```toml [dependencies] parquet2 = { version = "0.17", features = ["full"] } ``` -------------------------------- ### Configuring Parquet Write Options Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Customize output settings for Parquet writing using `write::WriteOptions`. This allows control over compression, encoding, and other parameters. ```rust write::WriteOptions ``` -------------------------------- ### Configure Parquet Write Features Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Specify compression codecs like 'snappy' and 'gzip' in your `Cargo.toml` to enable them for writing. For asynchronous operations, include the 'async' feature. ```toml [dependencies] parquet2 = { version = "0.17", features = ["snappy", "gzip"] } ``` ```toml parquet2 = { version = "0.17", features = ["async", "gzip", "snappy"] } ``` -------------------------------- ### Get Leaf Columns from SchemaDescriptor Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns a slice of all leaf `ColumnDescriptor`s within the schema. Useful for iterating over all individual columns. ```rust pub fn columns(&self) -> &[ColumnDescriptor] ``` -------------------------------- ### Get Total Byte Size of RowGroup Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the total uncompressed size in bytes for all columns within this row group. ```rust pub fn total_byte_size(&self) -> i64 ``` -------------------------------- ### Write Sidecar Metadata File Source: https://github.com/jorgecarleitao/parquet2/blob/main/guide/src/README.md Demonstrates writing a "sidecar" metadata file that consolidates metadata from multiple Parquet files, including statistics. This is useful when managing a collection of Parquet files. ```rust use parquet2::write::{FileWriter, RowGroupWriter}; use parquet2::metadata::Metadata; use parquet2::schema::types::SchemaDescriptor; use parquet2::page::DataPage; use parquet2::error::Result; use std::fs::File; fn main() -> Result<()> { // Placeholder for actual file writing logic // This example illustrates the concept of a sidecar file, not a complete implementation. // let mut file = File::create("sidecar.metadata")?; // let metadata: Metadata = ...; // Construct or read metadata // file.write_all(metadata.to_bytes().as_slice())?; println!("Sidecar metadata writing logic would go here."); Ok(()) } ``` -------------------------------- ### Get Schema Descriptor Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the schema descriptor from the FileMetaData. This is useful for understanding the structure of the data within the Parquet file. ```rust pub fn schema(&self) -> &SchemaDescriptor ``` ```rust let schema = metadata.schema(); let columns = schema.columns(); ``` -------------------------------- ### Configure Parquet Write Options Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Use WriteOptions to configure Parquet file writing, including statistics and version. ```rust pub struct WriteOptions { pub write_statistics: bool, pub version: Version, } ``` ```rust use parquet2::write::{WriteOptions, Version}; let options = WriteOptions { write_statistics: true, version: Version::V2, }; ``` -------------------------------- ### Create CompressedDictPage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Constructs a new compressed dictionary page. Use this for dictionary-encoded data that has been compressed. ```rust pub fn new( buffer: Vec, compression: Compression, uncompressed_page_size: usize, num_values: usize, is_sorted: bool, ) -> Self ``` -------------------------------- ### Typical Parquet File Structure Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Illustrates the common byte-level structure of a parquet file, including the magic bytes, row group data, metadata, and footer. ```text [4 bytes] PARQUET_MAGIC ("PAR1") [variable] Row Group 1 Data [variable] Row Group 2 Data [...] [variable] Metadata (TBrotlievel encoded) [4 bytes] Metadata size (little-endian i32) [4 bytes] PARQUET_MAGIC ("PAR1") ``` -------------------------------- ### Import Paths for Schema and Types Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Imports for working with Parquet schemas, primitive types, and native data types. ```rust use parquet2::{ schema::types::{ParquetType, PrimitiveType, PhysicalType}, metadata::ColumnDescriptor, types::NativeType, }; ``` -------------------------------- ### Get Top-Level Fields from SchemaDescriptor Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns a slice of the top-level fields within the schema. These represent the root elements of the schema structure. ```rust pub fn fields(&self) -> &[ParquetType] ``` -------------------------------- ### Create DictPage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Constructs a new uncompressed dictionary page. Use this when creating or working with dictionary-encoded data that is not compressed. ```rust pub fn new(buffer: Vec, num_values: usize, is_sorted: bool) -> Self ``` -------------------------------- ### Usage of ColumnDescriptor Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Demonstrates how to access and print information from a ColumnDescriptor. Requires a `column_meta` object with a `descriptor` method. ```rust let descriptor = column_meta.descriptor(); println!("Path: {}", descriptor.path_in_schema.join(".")); println!("Max def level: {}", descriptor.max_def_level); println!("Max rep level: {}", descriptor.max_rep_level); ``` -------------------------------- ### Get File Offset for Column Chunk Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the byte offset within the file where this column chunk's data begins. ```rust pub fn file_offset(&self) -> i64 ``` -------------------------------- ### Get Key-Value Metadata Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves optional user-defined key-value metadata associated with the Parquet file. This can be used to store custom information. ```rust pub fn key_value_metadata(&self) -> &Option> ``` -------------------------------- ### Enable Compression Features in Cargo.toml Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Specify compression codecs to enable in your Cargo.toml file. Use 'full' to enable all supported compression features. ```toml parquet2 = { version = "0.17", features = ["snappy", "gzip", "lz4", "zstd", "brotli"] } ``` ```toml parquet2 = { version = "0.17", features = ["full"] } ``` -------------------------------- ### Compress Data with Gzip Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/API_REFERENCE.md Compress input data using Gzip compression. Specify `CompressionOptions::Gzip(None)` for default Gzip settings. The compressed data is written to `output_buf`. ```rust use parquet2::compression::{compress, CompressionOptions}; let options = CompressionOptions::Gzip(None); compress(options, input_buf, &mut output_buf)?; ``` -------------------------------- ### Get Number of Values in Column Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the total count of values in the column. Note that this may differ from the number of rows for nested types. ```rust pub fn num_values(&self) -> i64 ``` -------------------------------- ### Supported Encodings for Writing Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Lists the supported encodings for writing Parquet pages, along with their corresponding modules and purposes. This information is crucial for selecting the appropriate encoding strategy. ```rust pub mod bitpacked; pub mod delta_bitpacked; pub mod delta_byte_array; pub mod delta_length_byte_array; pub mod hybrid_rle; pub mod plain_byte_array; pub mod uleb128; pub mod zigzag_leb128; ``` -------------------------------- ### Get File Path for Column Chunk Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the file path where the column data is stored, if it's in a separate file. The path is relative. ```rust pub fn file_path(&self) -> &Option ``` -------------------------------- ### Get Column Order Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Retrieves the sort order for a specific column by its index. Returns ColumnOrder::Undefined if the sort order is not available for the column. ```rust pub fn column_order(&self, i: usize) -> ColumnOrder ``` ```rust let order = metadata.column_order(0); ``` -------------------------------- ### into_thrift() Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Converts file metadata into its Thrift representation for writing. ```APIDOC ## into_thrift() ### Description Converts to Thrift representation for writing. ### Method Signature `pub fn into_thrift(self) -> parquet_format_safe::FileMetaData` ``` -------------------------------- ### Create CompressedDataPage Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Constructs a new compressed data page. Use this when you have data that has already been compressed and encoded. ```rust pub fn new( header: DataPageHeader, buffer: Vec, compression: Compression, uncompressed_page_size: usize, descriptor: Descriptor, rows: Option, ) -> Self ``` -------------------------------- ### Initialize FileWriter Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md The FileWriter struct is the main interface for writing complete Parquet files. ```rust pub struct FileWriter { // Writes a complete Parquet file } ``` -------------------------------- ### WriteOptions Struct Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_summary.md Configuration options for writing Parquet files. ```APIDOC ## WriteOptions Struct ### Description Configuration options for writing Parquet files. ### Fields - `write_statistics` (bool) - Whether to write statistics. - `version` (Version) - The Parquet version to use. ``` -------------------------------- ### Get Number of Values from DataPageHeader Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/types_and_pages.md Retrieves the number of values contained within a data page, regardless of whether it's a V1 or V2 header. ```rust pub fn num_values(&self) -> usize ``` -------------------------------- ### WriteOptions Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md Configuration options for writing Parquet files. Allows setting whether to write statistics and the Parquet version. ```APIDOC ## WriteOptions ### Description Configuration options for writing Parquet files. ### Fields - `write_statistics` (bool) - Required - Write min/max statistics and indexes. Defaults to true. - `version` (Version) - Required - Parquet version (V1 or V2). ### Usage Example ```rust use parquet2::write::{WriteOptions, Version}; let options = WriteOptions { write_statistics: true, version: Version::V2, }; ``` ``` -------------------------------- ### Import Paths for Writing Parquet Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Imports necessary for writing Parquet files, including file writers, options, and compression. ```rust use parquet2::{ write::{FileWriter, WriteOptions, Version}, page::CompressedPage, compression::CompressionOptions, metadata::FileMetaData, error::Result, }; ``` -------------------------------- ### InvalidParameter Error Example Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Illustrates how to return an InvalidParameter error when a parameter like page_size is invalid. This error is used when a function receives an input that violates its preconditions. ```rust if page_size == 0 { return Err(Error::InvalidParameter( "page_size must be greater than 0".into() )); } ``` -------------------------------- ### Wrap Compression Library Errors Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Errors from various compression libraries (Snappy, Gzip, Lz4) are wrapped into `Error::OutOfSpec`. ```rust - Snap (Snappy): `snap::Error` → `OutOfSpec` - Flate2 (Gzip): `std::io::Error` → `OutOfSpec` - Lz4: Library errors → `OutOfSpec` ``` -------------------------------- ### Enable Async Support in Cargo.toml Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Add the 'async' feature to your Cargo.toml to enable asynchronous operations for reading and writing parquet files. ```toml parquet2 = { version = "0.17", features = ["async"] } ``` -------------------------------- ### Import Path for Bloom Filters Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Import for the bloom filter module, used for efficient set membership testing. ```rust use parquet2::bloom_filter; ``` -------------------------------- ### Get Column Chunks for a Field Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/read_module.md Returns an iterator over column chunks associated with a given field name. Handles nested fields by matching the first component of the name. ```rust let row_group = &metadata.row_groups[0]; let columns = row_group.columns(); for col_meta in get_field_columns(columns, "my_field") { println!("Found column: {:?}", col_meta.descriptor()); } ``` -------------------------------- ### ZstdLevel struct and new method Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Defines the Zstandard compression level. Use the `new` method to create a ZstdLevel with a level between 1 and 22. Higher levels provide better compression. ```rust pub struct ZstdLevel(i32); pub fn new(level: i32) -> Self ``` -------------------------------- ### Async FileStreamer for Writing Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/write_module.md The asynchronous interface for writing Parquet files. Requires the 'async' feature to be enabled. ```rust #[cfg(feature = "async")] pub struct FileStreamer { // Async variant of FileWriter } ``` -------------------------------- ### Usage of BinaryStatistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Demonstrates how to access and print the min and max values from BinaryStatistics. Ensure the statistics object is correctly downcasted. ```rust let stats = stats .as_any() .downcast_ref::() .unwrap(); if let Some(min) = &stats.min_value { println!("Min: {}", String::from_utf8_lossy(min)); } if let Some(max) = &stats.max_value { println!("Max: {}", String::from_utf8_lossy(max)); } ``` -------------------------------- ### Getting Parquet Page Locations Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Use `read::read_pages_locations()` to determine the byte offsets of pages within a Parquet file. This can optimize reading by directly seeking to page data. ```rust read::read_pages_locations() ``` -------------------------------- ### Create ColumnChunkMetaData Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Constructor for creating a new ColumnChunkMetaData instance. ```rust pub fn new(column_chunk: ColumnChunk, column_descr: ColumnDescriptor) -> Self ``` -------------------------------- ### Stream Compressed Pages (Async) Source: https://github.com/jorgecarleitao/parquet2/blob/main/guide/src/README.md Get an asynchronous stream of compressed data pages for a column chunk using `parquet2::read::get_page_stream`. This is suitable for non-blocking processing of large datasets. ```rust use parquet2::read::{read_metadata_async, get_page_stream}; use tokio::fs::File; #[tokio::main] async fn main() { // Example usage: replace with actual file path and error handling let file = File::open("path/to/your/file.parquet").await.unwrap(); let file_metadata = read_metadata_async(file).await.unwrap(); if let Some(row_group) = file_metadata.row_groups.first() { if let Some(column_chunk) = row_group.columns().first() { let mut page_stream = get_page_stream(column_chunk, 0, 1024, None).await.unwrap(); // Adjust buffer size and other parameters as needed while let Some(page) = page_stream.next().await { println!("Processing async page..."); // Process each CompressedDataPage here } } } } ``` -------------------------------- ### Parallel Reading Pattern in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Demonstrates reading Parquet pages in the main thread (IO-bound) and processing them in a separate thread pool (CPU-bound) for performance optimization. Ensure proper error handling for decompression and deserialization. ```rust use parquet2::read::get_page_iterator; use std::thread; let column_meta = &metadata.row_groups[0].columns()[0]; // Read pages in main thread (IO-intensive) let compressed_pages: Vec<_> = get_page_iterator(column_meta, &mut file, None, vec![], 1024*1024)? .collect::>()?; // Process pages in thread pool (CPU-intensive) let handle = thread::spawn(move { for page in compressed_pages { // Decompress and deserialize let decompressed = decompress(...)?; // Process } }); handle.join().unwrap(); ``` -------------------------------- ### Handle Parquet2 Errors in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Match on the specific parquet2 Error enum variants to handle different error conditions gracefully. This example shows how to check for missing features, specification violations, and allocation issues. ```rust use parquet2::error::{Error, Feature}; match operation() { Ok(result) => { /* use result */ } Err(Error::FeatureNotActive(Feature::Snappy, _)) => { eprintln!("Enable snappy feature"); } Err(Error::OutOfSpec(msg)) => { eprintln!("Invalid parquet file: {}", msg); } Err(Error::WouldOverAllocate) => { eprintln!("Page too large"); } Err(e) => eprintln!("Error: {}", e), } ``` -------------------------------- ### Configure Parquet2 Dependencies with Features Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Configure Parquet2 dependencies in Cargo.toml to enable specific features like compression codecs or async support. The default features include snappy, gzip, lz4, zstd, brotli, and bloom_filter. ```toml [dependencies] # Minimal (uncompressed only) parquet2 = "0.17" # With compression support parquet2 = { version = "0.17", features = ["snappy", "gzip", "lz4", "zstd"] } # With async support parquet2 = { version = "0.17", features = ["async"] } # Everything parquet2 = { version = "0.17", features = ["full"] } ``` -------------------------------- ### try_from_thrift(metadata: parquet_format_safe::FileMetaData) Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Converts file metadata from its internal Thrift representation. ```APIDOC ## try_from_thrift(metadata: parquet_format_safe::FileMetaData) ### Description Converts from Thrift representation (internal format used in files). ### Method Signature `pub fn try_from_thrift(metadata: parquet_format_safe::FileMetaData) -> Result` ``` -------------------------------- ### Read Parquet File Metadata (Async) Source: https://github.com/jorgecarleitao/parquet2/blob/main/guide/src/README.md Utilize `parquet2::read::read_metadata_async` for asynchronous reads of Parquet file metadata, commonly used with `tokio::fs` for non-blocking IO operations. ```rust use parquet2::read::read_metadata_async; use tokio::fs::File; #[tokio::main] async fn main() { // Example usage: replace with actual file path and error handling let file = File::open("path/to/your/file.parquet").await.unwrap(); let file_metadata = read_metadata_async(file).await.unwrap(); println!("{:#?}", file_metadata); } ``` -------------------------------- ### Import Paths for Reading Parquet Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Imports required for reading Parquet file metadata, pages, and decompression. ```rust use parquet2::{ read::{read_metadata, get_page_iterator, decompress}, page::{Page, CompressedPage}, compression::Compression, metadata::FileMetaData, error::Result, }; ``` -------------------------------- ### Filter Parquet Row Groups with Statistics in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/README.md Filter Parquet row groups based on column statistics. This example shows how to extract statistics for an i32 column and check if a row group might contain values within a specific range. ```rust if let Some(result) = column_meta.statistics() { let stats = result?; let int_stats = stats.as_any() .downcast_ref::>() .unwrap(); if let (Some(min), Some(max)) = (int_stats.min_value, int_stats.max_value) { // Check if row group matches query } } ``` -------------------------------- ### Usage of SchemaDescriptor::columns Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Demonstrates iterating through all leaf columns in a schema and printing their names. Requires a `metadata` object with a `schema` method. ```rust let schema = metadata.schema(); for descriptor in schema.columns() { println!("Column: {}", descriptor.descriptor.field_info.name); } ``` -------------------------------- ### BrotliLevel struct and new method Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Defines the Brotli compression level. Use the `new` method to create a BrotliLevel with a level between 0 and 11. Higher levels provide better compression. ```rust pub struct BrotliLevel(u32); pub fn new(level: u32) -> Self ``` -------------------------------- ### Decompress Data with Specified Algorithm Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/read_module.md Use this function to decompress a buffer using a specified compression algorithm like Gzip. Ensure the output buffer is of the exact required size. The compression algorithm must be enabled. ```rust pub fn decompress( compression: Compression, input: &[u8], output: &mut [u8] ) -> Result<()> ``` ```rust use parquet2::read::decompress; use parquet2::compression::Compression; let mut output = vec![0u8; 1024]; decompress(Compression::Gzip, compressed_data, &mut output)?; ``` -------------------------------- ### Writing Parquet Files Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/API_REFERENCE.md Offers capabilities for writing Parquet files through the `write` module. ```APIDOC ## Writing Parquet Files ### Description Provides capabilities for writing Parquet files through the `write` module. ### Key Types - `FileWriter` — Main writer for creating parquet files - `FileStreamer` — Async variant (requires `async` feature) - `Version` — Parquet version (V1 or V2) - `WriteOptions` — Write configuration ``` -------------------------------- ### Usage of Display for Error Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/errors.md Demonstrates how to use the `Display` implementation of the `Error` type to print errors to stderr. ```rust match operation() { Ok(result) => { /* use result */ } Err(e) => eprintln!("Error: {}", e), // Uses Display impl } ``` -------------------------------- ### compress Function Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Compresses input data using the specified compression algorithm and options. The output is appended to the provided buffer. ```APIDOC ## compress Function ```rust pub fn compress( compression: CompressionOptions, input_buf: &[u8], output_buf: &mut Vec, ) -> Result<()> ``` Compresses data using the specified algorithm. ### Parameters: | Name | Type | Required | Description | |------|------|----------|-------------| | compression | `CompressionOptions` | Yes | Compression algorithm and options | | input_buf | `&[u8]` | Yes | Data to compress | | output_buf | `&mut Vec` | Yes | Buffer to append compressed data (not cleared first) | ### Return type: `Result<()>` ### Throws/rejects: | Error | Condition | |-------|-----------| | `FeatureNotActive` | Compression algorithm not enabled | | Compression errors | Data is invalid or too large | | `WouldOverAllocate` | Would exceed memory limits | **Important:** The output buffer is appended to, not replaced. Call `output_buf.clear()` before reusing for multiple compress calls. ### Usage: ```rust use parquet2::compression::{compress, CompressionOptions}; let mut output = Vec::new(); // Compress with Gzip compress( CompressionOptions::Gzip(None), b"hello world", &mut output )?; println!("Compressed {} bytes", output.len()); // For next compression, clear first output.clear(); compress( CompressionOptions::Snappy, b"more data", &mut output )?; ``` **Source:** `src/compression.rs:28` ``` -------------------------------- ### GzipLevel struct and new method Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Defines the Gzip compression level. Use the `new` method to create a GzipLevel with a level between 0 and 9. Higher levels provide better compression. The default level is typically 6. ```rust pub struct GzipLevel(u32); pub fn new(level: u32) -> Result ``` -------------------------------- ### Create PageReader in Rust Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/read_module.md Initializes a PageReader to sequentially read compressed pages for a column chunk. Requires a reader, column chunk metadata, a page filter, scratch space, and max page size. ```rust pub fn new( reader: R, column_chunk: &ColumnChunkMetaData, pages_filter: Arc bool>, scratch: Vec, max_page_size: usize ) -> Self ``` -------------------------------- ### Compress data using specified algorithm Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/compression_and_statistics.md Compresses data using the specified algorithm and options. The output buffer is appended to, not cleared. Ensure to clear the buffer before reusing it for subsequent compress calls. Handles `FeatureNotActive`, compression errors, and `WouldOverAllocate`. ```rust use parquet2::compression::{compress, CompressionOptions}; let mut output = Vec::new(); // Compress with Gzip compress( CompressionOptions::Gzip(None), b"hello world", &mut output )?; println!("Compressed {} bytes", output.len()); // For next compression, clear first output.clear(); compress( CompressionOptions::Snappy, b"more data", &mut output )?; ``` -------------------------------- ### schema() Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/metadata_structures.md Returns the schema descriptor for the file metadata. ```APIDOC ## schema() ### Description Returns the schema descriptor. ### Method Signature `pub fn schema(&self) -> &SchemaDescriptor` ### Usage ```rust let schema = metadata.schema(); let columns = schema.columns(); ``` ``` -------------------------------- ### Build Required Struct Schema Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/schema_types.md Constructs a Parquet schema for a struct containing a required i32 field and an optional string field. This demonstrates the creation of `ParquetType` instances with specified field information, logical types, and physical types. ```rust use parquet2::schema::types::*; // Build a schema for: required struct with int32 field and optional string field let field1 = ParquetType::PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "id".into(), repetition: Repetition::Required, id: Some(1), }, logical_type: None, converted_type: None, physical_type: PhysicalType::Int32, }); let field2 = ParquetType::PrimitiveType(PrimitiveType { field_info: FieldInfo { name: "name".into(), repetition: Repetition::Optional, id: Some(2), }, logical_type: Some(PrimitiveLogicalType::String), converted_type: None, physical_type: PhysicalType::ByteArray, }); let schema = ParquetType::new_root("schema".into(), vec![field1, field2]); ``` -------------------------------- ### Import Paths for Statistics Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/module_structure.md Imports for accessing and utilizing statistics within Parquet files. ```rust use parquet2::{ statistics::{Statistics, PrimitiveStatistics, BinaryStatistics}, schema::types::PhysicalType, }; ``` -------------------------------- ### Convert Page Header Type Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/QUICK_REFERENCE.md Use a match statement to handle different versions of DataPageHeader. ```rust use parquet2::page::DataPageHeader; match page_header { DataPageHeader::V1(header) => { /* V1-specific logic */ } DataPageHeader::V2(header) => { /* V2-specific logic */ } } ``` -------------------------------- ### Read Parquet File Metadata Source: https://github.com/jorgecarleitao/parquet2/blob/main/_autodocs/read_module.md Reads the complete file metadata by seeking to the end and parsing the footer. Requires a reader that supports seeking. Throws errors for small files, invalid footers, or IO issues. ```rust use parquet2::read::read_metadata; use std::fs::File; let mut file = File::open("example.parquet")?; let metadata = read_metadata(&mut file)?; println!("Version: {}", metadata.version); println!("Rows: {}", metadata.num_rows); ```