### Compress Data with LZ4 Compressor Source: https://context7.com/airlift/aircompressor/llms.txt Demonstrates basic block compression using the LZ4 algorithm. The `Compressor.create()` method automatically selects the optimal native or Java implementation. Ensure the output buffer is sized using `maxCompressedLength` to avoid truncation. ```java import io.airlift.compress.v3.Compressor; import io.airlift.compress.v3.lz4.Lz4Compressor; // Create a compressor (auto-selects native or Java implementation) Compressor compressor = Lz4Compressor.create(); // Prepare input data byte[] input = "Hello, World! This is some sample text to compress.".getBytes(); // Allocate output buffer with maximum possible compressed size byte[] output = new byte[compressor.maxCompressedLength(input.length)]; // Compress the data int compressedSize = compressor.compress(input, 0, input.length, output, 0, output.length); System.out.println("Original size: " + input.length); System.out.println("Compressed size: " + compressedSize); // Output: // Original size: 51 // Compressed size: 53 ``` -------------------------------- ### Configure AirCompressor System Properties Source: https://context7.com/airlift/aircompressor/llms.txt Set system properties to customize temporary directories or force pure Java implementations. Use the provided static methods to verify if native implementations are active. ```java // Set custom temporary directory for native library extraction // Useful when default temp directory is mounted with noexec System.setProperty("aircompressor.tmpdir", "/custom/tmp"); // Disable all native implementations (use pure Java only) System.setProperty("io.airlift.compress.v3.disable-native", "true"); // Check if native implementations are available boolean lz4Native = io.airlift.compress.v3.lz4.Lz4Native.isEnabled(); boolean snappyNative = io.airlift.compress.v3.snappy.SnappyNative.isEnabled(); boolean zstdNative = io.airlift.compress.v3.zstd.ZstdNative.isEnabled(); boolean xxhash3Native = io.airlift.compress.v3.xxhash.XxHash3Native.isEnabled(); System.out.println("LZ4 native: " + lz4Native); System.out.println("Snappy native: " + snappyNative); System.out.println("Zstd native: " + zstdNative); System.out.println("XXHash3 native: " + xxhash3Native); ``` -------------------------------- ### Implement Hadoop CompressionCodecs Source: https://context7.com/airlift/aircompressor/llms.txt Instantiate specific codec classes for direct integration with Hadoop framework file operations. Requires a Hadoop Configuration object to be set on the codec. ```java import io.airlift.compress.v3.zstd.ZstdCodec; import io.airlift.compress.v3.lz4.Lz4Codec; import io.airlift.compress.v3.snappy.SnappyCodec; import io.airlift.compress.v3.lzo.LzoCodec; import io.airlift.compress.v3.lzo.LzopCodec; import io.airlift.compress.v3.bzip2.BZip2Codec; import io.airlift.compress.v3.gzip.JdkGzipCodec; import io.airlift.compress.v3.deflate.JdkDeflateCodec; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.compress.CompressionCodec; // Create codec instances for Hadoop integration Configuration conf = new Configuration(); CompressionCodec zstdCodec = new ZstdCodec(); zstdCodec.setConf(conf); CompressionCodec lz4Codec = new Lz4Codec(); lz4Codec.setConf(conf); // Use with Hadoop file operations // FileSystem fs = FileSystem.get(conf); // FSDataOutputStream out = fs.create(new Path("/data/file.zst")); // CompressionOutputStream compressedOut = zstdCodec.createOutputStream(out); ``` -------------------------------- ### Configure LZ4 acceleration Source: https://context7.com/airlift/aircompressor/llms.txt Adjust the acceleration parameter to trade compression ratio for speed. Higher values result in faster compression. ```java import io.airlift.compress.v3.lz4.Lz4Compressor; // Create with custom acceleration (higher = faster but lower compression) // Default acceleration is 1, valid range depends on native implementation Lz4Compressor fastCompressor = Lz4Compressor.create(2); byte[] data = loadLargeDataset(); byte[] compressed = new byte[fastCompressor.maxCompressedLength(data.length)]; int compressedSize = fastCompressor.compress(data, 0, data.length, compressed, 0, compressed.length); ``` -------------------------------- ### Create Hadoop-Compatible Compression Streams Source: https://context7.com/airlift/aircompressor/llms.txt Utilize HadoopStreams implementations to create compression streams without direct Hadoop dependencies. Ensure streams are finished or closed properly to flush data. ```java import io.airlift.compress.v3.hadoop.HadoopInputStream; import io.airlift.compress.v3.hadoop.HadoopOutputStream; import io.airlift.compress.v3.hadoop.HadoopStreams; import io.airlift.compress.v3.zstd.ZstdHadoopStreams; import io.airlift.compress.v3.lz4.Lz4HadoopStreams; import io.airlift.compress.v3.snappy.SnappyHadoopStreams; import java.io.*; // Zstandard Hadoop streams HadoopStreams zstdStreams = new ZstdHadoopStreams(); System.out.println("Extension: " + zstdStreams.getDefaultFileExtension()); // .zst System.out.println("Codec: " + zstdStreams.getHadoopCodecName()); // org.apache.hadoop.io.compress.ZStandardCodec // Write compressed data try (OutputStream fileOut = new FileOutputStream("data" + zstdStreams.getDefaultFileExtension()); HadoopOutputStream compressedOut = zstdStreams.createOutputStream(fileOut)) { compressedOut.write("Hadoop-compatible compressed data".getBytes()); compressedOut.finish(); // Finish compression without closing underlying stream } // Read compressed data try (InputStream fileIn = new FileInputStream("data.zst"); HadoopInputStream compressedIn = zstdStreams.createInputStream(fileIn)) { byte[] buffer = new byte[1024]; int read = compressedIn.read(buffer); System.out.println(new String(buffer, 0, read)); compressedIn.resetState(); // Reset for reading next block } // LZ4 with custom configuration Lz4HadoopStreams lz4Streams = new Lz4HadoopStreams(true, 512 * 1024); // native enabled, 512KB buffer // Snappy Hadoop streams HadoopStreams snappyStreams = new SnappyHadoopStreams(); ``` -------------------------------- ### Compress and Decompress byte arrays with LZ4 Source: https://github.com/airlift/aircompressor/blob/master/README.md Demonstrates the byte array API for LZ4 compression and decompression. Ensure sufficient buffer sizes for compressed and uncompressed data. ```java byte[] data = ... Compressor compressor = new Lz4JavaCompressor(); byte[] compressed = new byte[compressor.maxCompressedLength(data.length)]; int compressedSize = compressor.compress(data, 0, data.length, compressed, 0, compressed.length); Decompressor decompressor = new Lz4JavaDecompressor(); byte[] uncompressed = new byte[data.length]; int uncompressedSize = decompressor.decompress(compressed, 0, compressedSize, uncompressed, 0, uncompressed.length); ``` -------------------------------- ### Deflate Compression and Decompression (Zlib-Compatible) Source: https://context7.com/airlift/aircompressor/llms.txt Offers Deflate block compression compatible with zlib/gzip libraries. Supports configurable compression levels from 1 (fastest) to 9 (maximum compression). ```java import io.airlift.compress.v3.deflate.DeflateCompressor; import io.airlift.compress.v3.deflate.DeflateDecompressor; // Create with default compression level DeflateCompressor compressor = DeflateCompressor.create(); // Or with specific compression level (1-9, where 9 is maximum compression) DeflateCompressor maxCompressor = DeflateCompressor.create(9); DeflateCompressor fastCompressor = DeflateCompressor.create(1); byte[] data = "Text data for deflate compression with maximum ratio".getBytes(); byte[] compressed = new byte[compressor.maxCompressedLength(data.length)]; int compressedSize = maxCompressor.compress(data, 0, data.length, compressed, 0, compressed.length); DeflateDecompressor decompressor = DeflateDecompressor.create(); byte[] restored = new byte[data.length]; decompressor.decompress(compressed, 0, compressedSize, restored, 0, restored.length); ``` -------------------------------- ### Zstandard Compression and Decompression Source: https://context7.com/airlift/aircompressor/llms.txt Utilizes the Zstandard algorithm for efficient compression and decompression. The `ZstdCompressor.create()` method supports default settings or specific compression levels when a native implementation is available. The decompressed size can be pre-determined using `getDecompressedSize`. ```java import io.airlift.compress.v3.zstd.ZstdCompressor; import io.airlift.compress.v3.zstd.ZstdDecompressor; // Create compressor with default settings (auto-selects native if available) ZstdCompressor compressor = ZstdCompressor.create(); // Or create with specific compression level (requires native implementation) // ZstdCompressor compressorWithLevel = ZstdCompressor.create(3); byte[] data = Files.readAllBytes(Path.of("large-file.txt")); byte[] compressed = new byte[compressor.maxCompressedLength(data.length)]; int compressedSize = compressor.compress(data, 0, data.length, compressed, 0, compressed.length); // Decompress ZstdDecompressor decompressor = ZstdDecompressor.create(); long decompressedSize = decompressor.getDecompressedSize(compressed, 0, compressedSize); byte[] output = new byte[(int) decompressedSize]; decompressor.decompress(compressed, 0, compressedSize, output, 0, output.length); ``` -------------------------------- ### Compress and decompress with SnappyCompressor Source: https://context7.com/airlift/aircompressor/llms.txt Provides memory-efficient block compression. Use getUncompressedLength to determine the buffer size required for decompression. ```java import io.airlift.compress.v3.snappy.SnappyCompressor; import io.airlift.compress.v3.snappy.SnappyDecompressor; SnappyCompressor compressor = SnappyCompressor.create(); SnappyDecompressor decompressor = SnappyDecompressor.create(); byte[] data = "Data to compress with Snappy algorithm".getBytes(); byte[] compressed = new byte[compressor.maxCompressedLength(data.length)]; int compressedSize = compressor.compress(data, 0, data.length, compressed, 0, compressed.length); // Get uncompressed length from compressed data header int uncompressedLength = decompressor.getUncompressedLength(compressed, 0); byte[] restored = new byte[uncompressedLength]; decompressor.decompress(compressed, 0, compressedSize, restored, 0, restored.length); ```