### Build Project with Maven Source: https://github.com/marcelmay/hfsa/blob/master/fsimage-generator/README.md This command builds the project using Maven. It packages the application, making it ready for execution. Ensure you have Maven installed and configured. ```bash mvn package ``` -------------------------------- ### Traverse HDFS from Specific Path using FsVisitor Source: https://context7.com/marcelmay/hfsa/llms.txt Demonstrates how to initiate an HDFS traversal starting from a specific directory path using the FsVisitor class. This is useful for analyzing particular sections of the filesystem. It allows for parallel traversal and defines actions for files, directories, and symlinks. ```java import de.m3y.hadoop.hdfs.hfsa.core.FsVisitor; // Traverse only the /data/warehouse directory and its subdirectories new FsVisitor.Builder() .parallel() .visit(fsImageData, new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { System.out.println("File in warehouse: " + path + "/" + inode.getName().toStringUtf8()); } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) { System.out.println("Directory: " + path + "/" + inode.getName().toStringUtf8()); } @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) { // Handle symlinks } }, "/data/warehouse"); // Start path ``` -------------------------------- ### Complete HDFS Analysis Example in Java Source: https://context7.com/marcelmay/hfsa/llms.txt This Java code demonstrates how to load an FSImage, traverse the filesystem, collect statistics like total files, directories, size, small files, user/group sizes, and file size distribution, and then print a formatted analysis report. It utilizes classes from the HFSA library for loading and visiting the filesystem image. ```java import de.m3y.hadoop.hdfs.hfsa.core.*; import de.m3y.hadoop.hdfs.hfsa.util.*; import org.apache.hadoop.fs.permission.PermissionStatus; import org.apache.hadoop.hdfs.server.namenode.FsImageProto; import java.io.RandomAccessFile; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.atomic.AtomicLong; public class HdfsAnalyzer { public static void main(String[] args) throws Exception { String fsImagePath = args[0]; // Load FSImage FsImageData fsImageData; try (RandomAccessFile file = new RandomAccessFile(fsImagePath, "r")) { fsImageData = new FsImageLoader.Builder() .parallel() .build() .load(file); } // Statistics collectors Map userSizes = new ConcurrentHashMap<>(); Map groupSizes = new ConcurrentHashMap<>(); SizeBucket sizeBucket = new SizeBucket(); AtomicLong totalFiles = new AtomicLong(0); AtomicLong totalDirs = new AtomicLong(0); AtomicLong totalSize = new AtomicLong(0); AtomicLong smallFiles = new AtomicLong(0); long smallFileThreshold = 2 * 1024 * 1024; // 2 MiB // Traverse filesystem new FsVisitor.Builder().parallel().visit(fsImageData, new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { FsImageProto.INodeSection.INodeFile file = inode.getFile(); long fileSize = FsUtil.getFileSize(file); totalFiles.incrementAndGet(); totalSize.addAndGet(fileSize); sizeBucket.add(fileSize); if (fileSize < smallFileThreshold) { smallFiles.incrementAndGet(); } PermissionStatus perms = fsImageData.getPermissionStatus(file.getPermission()); userSizes.computeIfAbsent(perms.getUserName(), k -> new AtomicLong(0)) .addAndGet(fileSize); groupSizes.computeIfAbsent(perms.getGroupName(), k -> new AtomicLong(0)) .addAndGet(fileSize); } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) { totalDirs.incrementAndGet(); } @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) {} }); // Print report System.out.println("=== HDFS Analysis Report ===\n"); System.out.printf("Total Files: %,d%n", totalFiles.get()); System.out.printf("Total Directories: %,d%n", totalDirs.get()); System.out.printf("Total Size: %s (%,d bytes)%n", IECBinary.format(totalSize.get()), totalSize.get()); System.out.printf("Small Files (< 2 MiB): %,d (%.1f%%)%n", smallFiles.get(), 100.0 * smallFiles.get() / totalFiles.get()); System.out.println("\n--- Top Users by Size ---"); userSizes.entrySet().stream() .sorted((a, b) -> Long.compare(b.getValue().get(), a.getValue().get())) .limit(10) .forEach(e -> System.out.printf(" %s: %s%n", e.getKey(), IECBinary.format(e.getValue().get()))); System.out.println("\n--- File Size Distribution ---"); for (int i = 0; i <= sizeBucket.findMaxNumBucket(); i++) { if (sizeBucket.getBucketCounter(i) > 0) { System.out.printf(" Bucket %d: %,d files%n", i, sizeBucket.getBucketCounter(i)); } } } } ``` -------------------------------- ### Traverse File System Hierarchy with Visitor Pattern (Java) Source: https://context7.com/marcelmay/hfsa/llms.txt Traverses all files, directories, and symlinks in a loaded FSImage using the FsVisitor interface. This example implements callback methods to collect statistics like total files, size, and directories. It supports both single-threaded and parallel execution strategies via the visitor builder. ```java import de.m3y.hadoop.hdfs.hfsa.core.FsImageData; import de.m3y.hadoop.hdfs.hfsa.core.FsVisitor; import de.m3y.hadoop.hdfs.hfsa.util.FsUtil; import org.apache.hadoop.fs.permission.PermissionStatus; import org.apache.hadoop.hdfs.server.namenode.FsImageProto; import java.util.concurrent.atomic.AtomicLong; import java.io.IOException; // Assume fsImageData is already loaded // FsImageData fsImageData = ...; AtomicLong totalFiles = new AtomicLong(0); AtomicLong totalSize = new AtomicLong(0); AtomicLong totalDirs = new AtomicLong(0); // Create visitor to collect statistics FsVisitor visitor = new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { totalFiles.incrementAndGet(); FsImageProto.INodeSection.INodeFile file = inode.getFile(); try { totalSize.addAndGet(FsUtil.getFileSize(file)); } catch (IOException e) { System.err.println("Error getting file size for " + path + ": " + e.getMessage()); return; // Skip processing this file if size cannot be retrieved } // Get file name and permissions String fileName = inode.getName().toStringUtf8(); PermissionStatus perms = fsImageData.getPermissionStatus(file.getPermission()); System.out.printf("File: %s/%s (owner=%s, size=%d bytes)%n", path, fileName, perms.getUserName(), FsUtil.getFileSize(file)); } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) { totalDirs.incrementAndGet(); String dirName = inode.getName().toStringUtf8(); System.out.printf("Directory: %s/%s%n", path, dirName); } @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) { String linkName = inode.getName().toStringUtf8(); String target = inode.getSymlink().getTarget().toStringUtf8(); System.out.printf("SymLink: %s/%s -> %s%n", path, linkName, target); } }; // Traverse entire filesystem with parallel processing try { new FsVisitor.Builder() .parallel() .visit(fsImageData, visitor); System.out.printf("Total: %d files, %d directories, %d bytes%n", totalFiles.get(), totalDirs.get(), totalSize.get()); } catch (IOException e) { System.err.println("Error during filesystem traversal: " + e.getMessage()); } ``` -------------------------------- ### Build HFSA from Source (Maven) Source: https://github.com/marcelmay/hfsa/blob/master/README.md This command builds the Hadoop FSImage Analyzer (HFSA) project from its source code using Maven. It performs a clean install of all project modules. Ensure Maven 3.9.x or compatible is installed. ```bash mvn clean install ``` -------------------------------- ### Load FSImage Files with Parallel Processing (Java) Source: https://context7.com/marcelmay/hfsa/llms.txt Loads an HDFS FSImage file into memory for analysis using FsImageLoader. This example demonstrates enabling parallel processing for improved performance on large images. It requires the `de.m3y.hadoop.hdfs.hfsa.core.FsImageLoader` class and handles potential IOExceptions during loading. ```java import de.m3y.hadoop.hdfs.hfsa.core.FsImageData; import de.m3y.hadoop.hdfs.hfsa.core.FsImageLoader; import java.io.RandomAccessFile; import java.io.IOException; // Load FSImage with parallel processing enabled try (RandomAccessFile file = new RandomAccessFile("/path/to/fsimage", "r")) { FsImageData fsImageData = new FsImageLoader.Builder() .parallel() // Enable multithreaded loading for better performance .build() .load(file); System.out.println("FSImage loaded successfully"); // fsImageData is now ready for traversal and analysis } catch (IOException e) { System.err.println("Failed to load FSImage: " + e.getMessage()); } ``` -------------------------------- ### Load and Visit FSImage Data with HFSA (Java) Source: https://github.com/marcelmay/hfsa/blob/master/lib/README.md Demonstrates how to load an FSImage file and traverse its inode hierarchy using the HFSA library. It shows parallel loading and visiting of files, directories, and symlinks. Dependencies include the HFSA library and standard Java I/O. ```java RandomAccessFile file = new RandomAccessFile("src/test/resources/fsi_small.img", "r"); FsImageData fsimageData = new FsImageLoader.Builder().parallel().build().load(file); new FsVisitor.Builder().parallel().visit(fsimageData,new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { System.out.println("Visiting file " + inode.getName().toStringUtf8()); // You can access size, permissions, ... } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) { System.out.println("Visiting directory " + inode.getName().toStringUtf8()); } @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) { System.out.println("Visiting sym link " + inode.getName().toStringUtf8()); } }, "/some/start/path"); ``` -------------------------------- ### Run FSImage Generator Source: https://github.com/marcelmay/hfsa/blob/master/fsimage-generator/README.md This command executes the FSImage generator JAR file. It can optionally take a system property to enable compression for the generated FSImage. The output shows the progress and final creation of the FSImage file. ```bash java [-Ddfs.image.compress=true] -jar target/hfsa-fsimage-generator-VERSION.jar ``` -------------------------------- ### File System Utilities with FsUtil Source: https://context7.com/marcelmay/hfsa/llms.txt Details the static utility methods provided by the FsUtil class for HDFS analysis. This includes checking INode types (file, directory, symlink), calculating file sizes (consumed and logical), retrieving replication factors, and obtaining storage policy information. It depends on Hadoop's FsImageProto and BlockStoragePolicy. ```java import de.m3y.hadoop.hdfs.hfsa.util.FsUtil; import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy; import org.apache.hadoop.hdfs.server.namenode.FsImageProto; FsImageProto.INodeSection.INode inode = fsImageData.getINodeFromPath("/data/myfile.txt"); // Type checking if (FsUtil.isFile(inode)) { FsImageProto.INodeSection.INodeFile file = inode.getFile(); // Get file size (sum of all block sizes) long fileSize = FsUtil.getFileSize(file); System.out.println("File size: " + fileSize + " bytes"); // Get consumed size (includes replication factor and erasure coding overhead) long consumedSize = FsUtil.getConsumedFileSize(file); System.out.println("Consumed size: " + consumedSize + " bytes"); // Get replication (handles erasure coding) int replication = FsUtil.getFileReplication(file); System.out.println("Replication: " + replication); // Get storage policy BlockStoragePolicy policy = FsUtil.getBlockStoragePolicy(file); System.out.println("Storage policy: " + policy.getName()); } else if (FsUtil.isDirectory(inode)) { System.out.println("This is a directory"); } else if (FsUtil.isSymlink(inode)) { System.out.println("This is a symlink"); String target = inode.getSymlink().getTarget().toStringUtf8(); System.out.println("Target: " + target); } ``` -------------------------------- ### List Paths with Filters (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt Lists all files and directories recursively from a given path, with options to filter by user and path patterns. It's analogous to a recursive `ls` command. ```bash # List all paths for specific directories, filtered by user ./hfsa-tool -fun="m.*" -p "/test3","/test1" /path/to/fsimage path # Output example: # Path report (paths=[/test3, /test1], user=~m.*) : # ------------------------------------------------- # # 8 files, 4 directories and 0 symlinks # # drwxr-xr-x mm supergroup /test1/test1 # drwxr-xr-x mm supergroup /test3/foo # drwxr-xr-x mm supergroup /test3/foo/bar # -rw-r--r-- mm nobody /test3/foo/bar/test_20MiB.img # -rw-r--r-- mm supergroup /test3/foo/bar/test_2MiB.img # ... ``` -------------------------------- ### Generate HDFS Summary Report (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt The `hfsa-tool` command-line utility generates comprehensive HDFS usage reports from FSImage files. Running it without a subcommand provides a default summary report. Options allow sorting, filtering by user, and specifying paths. ```bash # Download and extract the tool wget https://repo1.maven.org/maven2/de/m3y/hadoop/hdfs/hfsa/hfsa-tool/1.3.12/hfsa-tool-1.3.12-bin.zip unzip hfsa-tool-1.3.12-bin.zip # Run summary report (default command) ./hfsa-tool-1.3.12/bin/hfsa-tool /path/to/fsimage # Output example: # HDFS Summary : / # ---------------- # # #Groups | #Users | #Directories | #Symlinks | #Files | Size [MB] | #Blocks | File Size Buckets # | | | | | | | 0 B 1 MiB 2 MiB 4 MiB ... # ------------------------------------------------------------------------------------------ # 3 | 3 | 8 | 0 | 11 | 331 | 12 | 0 2 1 2 ... # # By group: ... # By user: ... # Sort by file count instead of size ./hfsa-tool /path/to/fsimage summary --sort fc # Filter by user name pattern (regex) ./hfsa-tool -fun="admin.*" /path/to/fsimage # Analyze specific paths ./hfsa-tool -p "/data/warehouse","/user" /path/to/fsimage # Enable verbose/debug output ./hfsa-tool -v /path/to/fsimage # verbose ./hfsa-tool -vv /path/to/fsimage # debug ``` -------------------------------- ### Generate FSImage with Maven (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt Builds the FSImage generator using Maven. This tool creates synthetic FSImage files for testing and benchmarking purposes. ```bash # Build the generator cd fsimage-generator mvn package ``` -------------------------------- ### Analyze File Size Distribution with SizeBucket (Java) Source: https://context7.com/marcelmay/hfsa/llms.txt The SizeBucket class categorizes file sizes into predefined buckets for analyzing distribution patterns. It's used during file system traversal to aggregate file size counts. The results can be computed into upper borders and counts for reporting. ```java import de.m3y.hadoop.hdfs.hfsa.util.SizeBucket; import de.m3y.hadoop.hdfs.hfsa.util.FsUtil; SizeBucket sizeBucket = new SizeBucket(); // Add file sizes during traversal new FsVisitor.Builder().parallel().visit(fsImageData, new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { long fileSize = FsUtil.getFileSize(inode.getFile()); sizeBucket.add(fileSize); } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) {} @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) {} }); // Get results long[] bucketUpperBorders = sizeBucket.computeBucketUpperBorders(); long[] bucketCounts = sizeBucket.get(); System.out.println("File Size Distribution:"); for (int i = 0; i <= sizeBucket.findMaxNumBucket(); i++) { String sizeLabel = (i == 0) ? "0 B" : "< " + (bucketUpperBorders[i] / 1024 / 1024) + " MiB"; System.out.printf(" %s: %d files%n", sizeLabel, sizeBucket.getBucketCounter(i)); } ``` -------------------------------- ### Run FSImage Generator (Java) Source: https://context7.com/marcelmay/hfsa/llms.txt Executes the FSImage generator JAR file to create test FSImage data. It can be run with default settings or with compression enabled. ```java # Run with default settings (creates ~800 directories, ~210k files) java -jar target/hfsa-fsimage-generator-1.3.12-SNAPSHOT.jar # Enable compression java -Ddfs.image.compress=true -jar target/hfsa-fsimage-generator-1.3.12-SNAPSHOT.jar # Output example: # Max depth = 5, max width = 2, files-factor = 10 # Generates 806 dirs (depth up to 5) and 209560 files # Progress: 100 directories and 26000 files... # ... # Created new FSImage containing meta data for 806 directories and 209560 files # FSImage path : /path/to/fsimage.img ``` -------------------------------- ### Add HFSA Library as Maven Dependency Source: https://context7.com/marcelmay/hfsa/llms.txt This XML snippet shows how to include the HFSA library in your Maven project's pom.xml file. Ensure you use the latest compatible version of the library. ```xml de.m3y.hadoop.hdfs.hfsa hfsa-lib 1.3.12 ``` -------------------------------- ### Load and Traverse HDFS fsimage using HFSA Library (Java) Source: https://github.com/marcelmay/hfsa/blob/master/README.md This Java code snippet demonstrates how to load an HDFS fsimage file into memory using HFSA's FsImageLoader and then traverse the file hierarchy using a FsVisitor. It shows how to access file and directory details like names, permissions, and file sizes. Dependencies include HFSA library and standard Java I/O. ```java RandomAccessFile file = new RandomAccessFile("src/test/resources/fsi_small.img", "r"); // Load file into memory FsImageData fsimageData = new FsImageLoader.Builder() .parallel().build() .load(file); // Traverse file hierarchy new FsVisitor.Builder() .parallel() .visit(fsimageData, new FsVisitor() { @Override public void onFile(FsImageProto.INodeSection.INode inode, String path) { // Do something String fileName = ("/".equals(path) ? path : path + '/') + inode.getName().toStringUtf8(); System.out.println(fileName); FsImageProto.INodeSection.INodeFile f = inode.getFile(); PermissionStatus p = loader.getPermissionStatus(f.getPermission()); ... } @Override public void onDirectory(FsImageProto.INodeSection.INode inode, String path) { // Do something final String dirName = ("/".equals(path) ? path : path + '/') + inode.getName().toStringUtf8(); System.out.println("Directory : " + fileName); FsImageProto.INodeSection.INodeDirectory d = inode.getDirectory(); PermissionStatus p = loader.getPermissionStatus(d.getPermission()); ... } @Override public void onSymLink(FsImageProto.INodeSection.INode inode, String path) { // Do something } } ); ``` -------------------------------- ### Display HDFS INode Details (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt This command allows displaying detailed information about specific HDFS inodes, identified by their path or inode ID. The output includes type, ID, name, and specific attributes like modification time, permissions, and block information. ```bash # Show inode details for paths ./hfsa-tool /path/to/fsimage inode "/test3" "/test3/test_160MiB.img" # Output example: # type: DIRECTORY # id: 16388 # name: "test3" # directory { # modificationTime: 1497734744891 # nsQuota: 18446744073709551615 # dsQuota: 18446744073709551615 # permission: 1099511759341 # } # # type: FILE # id: 16402 # name: "test_160MiB.img" # file { # replication: 1 # modificationTime: 1497734744886 # accessTime: 1497734743534 # preferredBlockSize: 134217728 # permission: 5497558401444 # } ``` -------------------------------- ### Query INodes by Path using FsImageData Source: https://context7.com/marcelmay/hfsa/llms.txt Explains how to use the FsImageData class to query HDFS metadata. It covers checking path existence, retrieving INodes by path, listing directory children (with and without filters), and retrieving file details like size, blocks, and permissions. Dependencies include Hadoop's FsImageProto and FsUtil. ```java import de.m3y.hadoop.hdfs.hfsa.core.FsImageData; import de.m3y.hadoop.hdfs.hfsa.util.FsUtil; import org.apache.hadoop.fs.permission.PermissionStatus; import org.apache.hadoop.hdfs.server.namenode.FsImageProto; import java.util.List; // Check if path exists if (fsImageData.hasINode("/data/important")) { System.out.println("Path exists!"); } // Get INode for a specific path FsImageProto.INodeSection.INode inode = fsImageData.getINodeFromPath("/data/myfile.txt"); System.out.println("Type: " + inode.getType()); // FILE, DIRECTORY, or SYMLINK System.out.println("Name: " + inode.getName().toStringUtf8()); // Get file details if (FsUtil.isFile(inode)) { FsImageProto.INodeSection.INodeFile file = inode.getFile(); System.out.println("Size: " + FsUtil.getFileSize(file) + " bytes"); System.out.println("Blocks: " + file.getBlocksCount()); System.out.println("Replication: " + FsUtil.getFileReplication(file)); } // Get permission status PermissionStatus perms = fsImageData.getPermissionStatus(inode); System.out.println("Owner: " + perms.getUserName()); System.out.println("Group: " + perms.getGroupName()); System.out.println("Permission: " + perms.getPermission()); // List child directories List childDirs = fsImageData.getChildDirectories("/data"); for (String childPath : childDirs) { System.out.println("Child directory: " + childPath); } // List child directories with filter List filteredDirs = fsImageData.getChildDirectories("/data", inode -> inode.getName().toStringUtf8().startsWith("test")); // Get files in directory List files = fsImageData.getFileINodesInDirectory("/data"); for (FsImageProto.INodeSection.INode fileNode : files) { System.out.println("File: " + fileNode.getName().toStringUtf8()); } // Check if directory has children boolean hasChildren = fsImageData.hasChildren("/data"); int numChildren = fsImageData.getNumChildren(inode); ``` -------------------------------- ### Generate HDFS Small Files Report (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt This report identifies directories with a high concentration of small files, a common HDFS performance issue. The default threshold is files smaller than 2 MiB. Options allow customizing the size limit and filtering by user. ```bash # Report small files (default: < 2 MiB) ./hfsa-tool /path/to/fsimage smallfiles # Report files smaller than 3 MiB, filtered by user ./hfsa-tool -fun="m.*" /path/to/fsimage smallfiles --fsl 3MiB # Output example: # Small files report (< 3 MiB) # # Overall small files : 4 # User (filtered) small files : 3 # # #Small files | Path (top 10) # ------------------------------ # 4 | / # 3 | /test3 # 2 | /test3/foo # 1 | /test3/foo/bar # # Username | #Small files # ----------------------- # mm | 3 # Customize hotspot limit ./hfsa-tool /path/to/fsimage smallfiles --uphl 20 ``` -------------------------------- ### Format Bytes to Human-Readable Size (Java) Source: https://context7.com/marcelmay/hfsa/llms.txt The IECBinary utility class converts byte values into human-readable strings using IEC binary prefixes (KiB, MiB, GiB, etc.). It also supports parsing human-readable strings back into byte values. This is useful for displaying file sizes in a user-friendly format. ```java import de.m3y.hadoop.hdfs.hfsa.util.IECBinary; // Format bytes to human-readable string long bytes = 1536000L; String formatted = IECBinary.format(bytes); // "1 MiB" System.out.println(IECBinary.format(1024)); // "1 KiB" System.out.println(IECBinary.format(1048576)); // "1 MiB" System.out.println(IECBinary.format(1073741824L)); // "1 GiB" System.out.println(IECBinary.format(1099511627776L)); // "1 TiB" // Parse human-readable string to bytes long parsed = IECBinary.parse("2 MiB"); // 2097152 long parsed2 = IECBinary.parse("512 KiB"); // 524288 long parsed3 = IECBinary.parse("100 GiB"); // 107374182400 ``` -------------------------------- ### Report User Usage with Age Filter (Bash) Source: https://context7.com/marcelmay/hfsa/llms.txt Finds and reports storage locations with the highest usage for a specific user. It supports an optional filter for files modified before a certain date. ```bash # Find top storage locations for user 'mm' with files older than 60 days ./hfsa-tool /path/to/fsimage uu -a 60d mm # Output example: # Size report (user=mm, start dir=/, last modification older 2021-05-12T23:49:44.203) # # / | 172 MiB # /test3 | 172 MiB # /test3/foo | 171 MiB # /test3/foo/bar | 151 MiB ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.