### ls get directory information example Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.ls with include_toplevel=True and include_children=False, returning only the directory itself. ```python >>> list(client.ls(["/source"], include_toplevel=True, include_children=False)) [{'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}] ``` -------------------------------- ### HAClient example: configure namenodes and list root Source: https://snakebite.readthedocs.io/en/latest/client.html Sets up a high-availability client with two namenodes and trash enabled. The HAClient is backwards compatible with the vanilla Client and can also be used for non-HA clusters. The example lists the root directory and prints each entry. ```python >>> from snakebite.client import HAClient >>> from snakebite.namenode import Namenode >>> n1 = Namenode("namenode1.mydomain", 8020) >>> n2 = Namenode("namenode2.mydomain", 8020) >>> client = HAClient([n1, n2], use_trash=True) >>> for x in client.ls(['/']): ... print x ``` -------------------------------- ### AutoConfigClient example: list root directory Source: https://snakebite.readthedocs.io/en/latest/client.html Creates an AutoConfigClient that reads HADOOP_HOME/conf/hdfs-site.xml and core-site.xml to locate the namenode. The default hadoop_version is 9, which can be overridden via the constructor parameter. The example lists the root directory and prints each entry. ```python >>> from snakebite.client import AutoConfigClient >>> client = AutoConfigClient() >>> for x in client.ls(['/']): ... print x ``` -------------------------------- ### Get server defaults with client.serverdefaults() Source: https://snakebite.readthedocs.io/en/latest/client.html Example output of calling client.serverdefaults() without arguments. Returns a dictionary with keys: blockSize, bytesPerChecksum, writePacketSize, replication, fileBufferSize, encryptDataTransfer, trashInterval, checksumType. ```python >>> client.serverdefaults() [{'writePacketSize': 65536, 'fileBufferSize': 4096, 'replication': 1, 'bytesPerChecksum': 512, 'trashInterval': 0L, 'blockSize': 134217728L, 'encryptDataTransfer': False, 'checksumType': 2}] ``` -------------------------------- ### ls directory listing example Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.ls(["/"]) with default parameters, listing directory contents with file metadata. ```python >>> list(client.ls(["/"])) [{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317325431L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/build'}, {'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317326510L, 'block_replication': 1, 'modification_time': 1367317326522L, 'length': 100L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/index.asciidoc'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}] ``` -------------------------------- ### ls file listing example Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.ls(["/Makefile"]) for a single file, returning only that file's metadata. ```python >>> list(client.ls(["/Makefile"])) [{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}] ``` -------------------------------- ### du directory only example Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.du with include_toplevel=True and include_children=False, returning only the top-level path. ```python >>> list(client.du(['/'], include_toplevel=True, include_children=False)) [{'path': '/', 'length': 260185L}] ``` -------------------------------- ### Get filesystem information with df() Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.df() as a dictionary containing used, capacity, under_replicated, missing_blocks, filesystem, remaining, and corrupt_blocks. ```python >>> client.df() {'used': 491520L, 'capacity': 120137519104L, 'under_replicated': 0L, 'missing_blocks': 0L, 'filesystem': 'hdfs://localhost:8020', 'remaining': 19669295104L, 'corrupt_blocks': 0L} ``` -------------------------------- ### Snakebite configuration file format Source: https://snakebite.readthedocs.io/en/latest/cli.html Example of a JSON-formatted configuration file for Snakebite, which can be placed in ~/.snakebiterc or /etc/snakebiterc. ```json { "config_version": 2, "skiptrash": true, "namenodes": [ {"host": "mynamenode1", "port": 8020, "version": 9}, {"host": "mynamenode2", "port": 8020, "version": 9} ] } ``` -------------------------------- ### du children example Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.du(['/']) with default parameters, listing child paths and their lengths. ```python >>> list(client.du(['/'])) [{'path': '/Makefile', 'length': 6783L}, {'path': '/build', 'length': 244778L}, {'path': '/index.asciidoc', 'length': 100L}, {'path': '/source', 'length': 8524L}] ``` -------------------------------- ### MiniCluster usage without snakebite.client Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Shows how to instantiate MiniCluster and call ls() without using the snakebite client. The cluster is started at instantiation. ```python >>> from snakebite.minicluster import MiniCluster >>> cluster = MiniCluster("/path/to/test/files") >>> ls_output = cluster.ls(["/"]) ``` -------------------------------- ### Stat a file with client.stat() Source: https://snakebite.readthedocs.io/en/latest/client.html Example output of calling client.stat(['/index.asciidoc']). Returns a dictionary with file metadata such as blocksize, owner, length, access_time, group, permission, file_type, path, modification_time, and block_replication. ```python >>> client.stat(['/index.asciidoc']) {'blocksize': 134217728L, 'owner': u'wouter', 'length': 100L, 'access_time': 1367317326510L, 'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'path': '/index.asciidoc', 'modification_time': 1367317326522L, 'block_replication': 1} ``` -------------------------------- ### Instantiate Client and list root directory Source: https://snakebite.readthedocs.io/en/latest/client.html Shows how to create a Client instance and consume the generator returned by ls() to list the root directory. Note that many methods return generators, so they must be consumed to execute. ```python >>> from snakebite.client import Client >>> client = Client("localhost", 8020, use_trash=False) >>> for x in client.ls(['/']): ... print x ``` -------------------------------- ### mkdir Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Creates a directory in the mini cluster, with optional extra arguments. ```APIDOC ## Function: mkdir ### Description Create a directory in the mini cluster. ### Parameters - **src** (string) - Required - The path of the directory to create. - **extra_args** (list) - Optional - Additional hadoop command arguments. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### Client() Source: https://snakebite.readthedocs.io/en/latest/client.html Creates a pure Python HDFS client instance with configurable connection and authentication options. ```APIDOC ## Client() ### Description Creates a pure Python HDFS client instance. ### Constructor `Client(host, port=8020, hadoop_version=9, use_trash=False, effective_user=None, use_sasl=False, hdfs_namenode_principal=None, sock_connect_timeout=10000, sock_request_timeout=10000, use_datanode_hostname=False)` ### Parameters - **host** (string) - Required - Hostname or IP address of the NameNode - **port** (int) - Optional - RPC Port of the NameNode (default: 8020) - **hadoop_version** (int) - Optional - What hadoop protocol version should be used (default: 9) - **use_trash** (boolean) - Optional - Use a trash when removing files. - **effective_user** (string) - Optional - Effective user for the HDFS operations (default: None - current user) - **use_sasl** (boolean) - Optional - Use SASL authentication or not - **hdfs_namenode_principal** (string) - Optional - Kerberos principal to use for HDFS - **sock_connect_timeout** (int) - Optional - Socket connection timeout in seconds - **sock_request_timeout** (int) - Optional - Request timeout in seconds - **use_datanode_hostname** (boolean) - Optional - Use hostname instead of IP address to communicate with datanodes ### Example ``` >>> from snakebite.client import Client >>> client = Client("localhost", 8020, use_trash=False) ``` ``` -------------------------------- ### ls Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Lists files in a directory of the mini cluster, supporting extra arguments. ```APIDOC ## Function: ls ### Description List files in a directory in the mini cluster. ### Parameters - **src** (string) - Required - The source path to list files from. - **extra_args** (list) - Optional - Additional hadoop command arguments. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### touchz(paths, replication=None, blocksize=None) Source: https://snakebite.readthedocs.io/en/latest/client.html Creates a zero-length file or updates the timestamp on an existing zero-length file. ```APIDOC ## touchz(paths, replication=None, blocksize=None) ### Description Creates a zero-length file or updates the timestamp on an existing zero-length file. ### Method touchz ### Parameters - **paths** (list) - Required - Paths to create or update - **replication** (int) - Optional - Replication factor for the new file. - **blocksize** (int) - Optional - Block size (in bytes) for the new file. ### Returns A generator that yields dictionaries. ``` -------------------------------- ### mkdir(paths, create_parent=False, mode=493) Source: https://snakebite.readthedocs.io/en/latest/client.html Creates directories at the specified paths. The function accepts a list of paths and optional flags for creating parent directories and setting the mode. ```APIDOC ## mkdir(paths, create_parent=False, mode=493) ### Description Creates a directory. ### Method N/A (Python function) ### Endpoint N/A (Python function) ### Parameters - **paths** (list of strings) - Required - Paths to create - **create_parent** (boolean) - Optional - Also create the parent directories - **mode** (int) - Optional - Mode the directory should be created with ### Returns A generator that yields dictionaries. ``` -------------------------------- ### Count files in a path Source: https://snakebite.readthedocs.io/en/latest/client.html Shows the output of client.count(['/']) as a list of dictionaries containing spaceConsumed, quota, spaceQuota, length, directoryCount, path, and fileCount. ```python >>> list(client.count(['/'])) [{'spaceConsumed': 260185L, 'quota': 2147483647L, 'spaceQuota': 18446744073709551615L, 'length': 260185L, 'directoryCount': 9L, 'path': '/', 'fileCount': 34L}] ``` -------------------------------- ### put Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Uploads a file to the mini cluster from the test files path provided during construction. ```APIDOC ## Function: put ### Description Uploads a file to HDFS. Takes a source path and a destination path, with the file sourced from `testfiles_path` provided in the constructor. ### Parameters - **src** (string) - Required - The path to the local file to upload. - **dst** (string) - Required - The destination path in HDFS. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### serverdefaults(force_reload=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Retrieves the default configuration values from the HDFS server, with optional caching and forced reload. ```APIDOC ## serverdefaults(force_reload=False) ### Description Gets the HDFS server defaults, caching the results. If force_reload is True or no cached results exist, it queries the server for default parameter values. ### Method serverdefaults ### Parameters - **force_reload** (bool) - Optional - If True, reload the server defaults even if they are already cached. Default is False. ### Returns A dictionary with keys: blockSize, bytesPerChecksum, writePacketSize, replication, fileBufferSize, encryptDataTransfer, trashInterval, checksumType. ### Example ``` >>> client.serverdefaults() [{'writePacketSize': 65536, 'fileBufferSize': 4096, 'replication': 1, 'bytesPerChecksum': 512, 'trashInterval': 0L, 'blockSize': 134217728L, 'encryptDataTransfer': False, 'checksumType': 2}] ``` ``` -------------------------------- ### df() Source: https://snakebite.readthedocs.io/en/latest/client.html Retrieves filesystem information from the HDFS cluster. ```APIDOC ## df() ### Description Get FS information. ### Returns A dictionary. ### Example ``` >>> client.df() {'used': 491520L, 'capacity': 120137519104L, 'under_replicated': 0L, 'missing_blocks': 0L, 'filesystem': 'hdfs://localhost:8020', 'remaining': 19669295104L, 'corrupt_blocks': 0L} ``` ``` -------------------------------- ### df Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Performs the hadoop df command on a specified path in the mini cluster. ```APIDOC ## Function: df ### Description Perform `df` on a path in the mini cluster to display disk usage. ### Parameters - **src** (string) - Required - The path on which to perform the df operation. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### test(path, exists=False, directory=False, zero_length=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Checks various properties of a path, such as existence, directory status, and zero length. ```APIDOC ## test(path, exists=False, directory=False, zero_length=False) ### Description Tests whether a path exists, is a directory, or has zero length. The directory and zero_length checks are AND'd together. ### Method test ### Parameters - **path** (string) - Required - Path to test - **exists** (boolean) - Optional - Check if the path exists. Default is False. - **directory** (boolean) - Optional - Check if the path is a directory. Default is False. - **zero_length** (boolean) - Optional - Check if the path is zero-length. Default is False. ### Returns A boolean indicating whether the specified conditions are met. ``` -------------------------------- ### ls(paths, recurse=False, include_toplevel=False, include_children=True) Source: https://snakebite.readthedocs.io/en/latest/client.html Lists file information for the given paths. The function returns a generator of dictionaries containing file metadata. ```APIDOC ## ls(paths, recurse=False, include_toplevel=False, include_children=True) ### Description Issues 'ls' command and returns a list of maps that contain file information. ### Method N/A (Python function) ### Endpoint N/A (Python function) ### Parameters - **paths** (list) - Required - Paths to list - **recurse** (boolean) - Optional - Recursive listing - **include_toplevel** (boolean) - Optional - Include the given path in the listing. If the path is a file, include_toplevel is always True. - **include_children** (boolean) - Optional - Include child nodes in the listing. ### Returns A generator that yields dictionaries. ### Example (Directory listing) ``` >>> list(client.ls(["/"])) [{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317325431L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/build'}, {'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317326510L, 'block_replication': 1, 'modification_time': 1367317326522L, 'length': 100L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/index.asciidoc'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}] ``` ### Example (File listing) ``` >>> list(client.ls(["/Makefile"])) [{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}] ``` ### Example (Get directory information) ``` >>> list(client.ls(["/source"], include_toplevel=True, include_children=False)) [{'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}] ``` ``` -------------------------------- ### cat(paths, check_crc=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Fetches file contents from HDFS and yields them as strings. ```APIDOC ## cat(paths, check_crc=False) ### Description Fetch all files that match the source file pattern and display their content on stdout. ### Parameters - **paths** (list of strings) - Required - Paths to display - **check_crc** (boolean) - Optional - Check for checksum errors ### Returns A generator that yields strings. ``` -------------------------------- ### exists Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Checks whether a given path exists in the mini cluster. ```APIDOC ## Function: exists ### Description Return True if the specified path exists, False if it doesn't. ### Parameters - **path** (string) - Required - The path to check for existence. ### Returns A boolean value indicating whether the path exists. ``` -------------------------------- ### Snakebite CLI usage and command reference Source: https://snakebite.readthedocs.io/en/latest/cli.html General syntax for the snakebite command, including available options and a list of supported HDFS commands. ```bash snakebite [general options] cmd [arguments] general options: -D --debug Show debug information -V --version Hadoop protocol version (default:9) -h --help show help -j --json JSON output -n --namenode namenode host -p --port namenode RPC port (default: 8020) -v --ver Display snakebite version commands: cat [paths] copy source paths to stdout chgrp [paths] change group chmod [paths] change file mode (octal) chown [paths] change owner copyToLocal [paths] dst copy paths to local file system destination count [paths] display stats for paths df display fs stats du [paths] display disk usage statistics get file dst copy files to local file system destination getmerge dir dst concatenates files in source dir into destination local file ls [paths] list a path mkdir [paths] create directories mkdirp [paths] create directories and their parents mv [paths] dst move paths to destination rm [paths] remove paths rmdir [dirs] delete a directory serverdefaults show server information setrep [paths] set replication factor stat [paths] stat information tail path display last kilobyte of the file to stdout test path test a path text path [paths] output file in text format touchz [paths] creates a file of zero length usage show cmd usage to see command-specific options use: snakebite [cmd] --help ``` -------------------------------- ### copyToLocal(paths, dst, check_crc=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Copies files from HDFS to the local filesystem. ```APIDOC ## copyToLocal(paths, dst, check_crc=False) ### Description Copy files that match the file source pattern to the local name. Source is kept. When copying multiple files, the destination must be a directory. ### Parameters - **paths** (list of strings) - Required - Paths to copy - **dst** (string) - Required - Destination path - **check_crc** (boolean) - Optional - Check for checksum errors ### Returns A generator that yields strings. ``` -------------------------------- ### MiniCluster usage with snakebite.client Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Shows how to use MiniCluster with the snakebite Client. The client connects to localhost on the cluster's port, and then ls() is called on the client. ```python >>> from snakebite.minicluster import MiniCluster >>> from snakebite.client import Client >>> cluster = MiniCluster("/path/to/test/files") >>> client = Client('localhost', cluster.port) >>> ls_output = client.ls(["/"]) ``` -------------------------------- ### setrep(paths, replication, recurse=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Sets the replication factor for one or more paths. Optionally applies the setting recursively. ```APIDOC ## setrep(paths, replication, recurse=False) ### Description Sets the replication factor for the specified paths. Optionally applies the replication factor recursively to subdirectories. ### Method setrep ### Parameters - **paths** (list) - Required - Paths to set replication factor for - **replication** (int) - Required - Replication factor - **recurse** (boolean) - Optional - If True, apply replication factor recursively. Default is False. ### Returns A generator that yields dictionaries. ``` -------------------------------- ### text(paths, check_crc=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Displays the contents of a file in text format, supporting gzip and bzip2 compressed files. ```APIDOC ## text(paths, check_crc=False) ### Description Outputs the contents of a file in text format. Supported formats are gzip and bzip2. ### Method text ### Parameters - **paths** (list of strings) - Required - Paths to display - **check_crc** (boolean) - Optional - If True, check for checksum errors. Default is False. ### Returns A generator that yields strings. ``` -------------------------------- ### du Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Performs the du command on a path in the mini cluster, allowing extra hadoop arguments. ```APIDOC ## Function: du ### Description Perform `du` on a path in the mini cluster to show disk usage details. ### Parameters - **src** (string) - Required - The path on which to perform the du operation. - **extra_args** (list) - Optional - Additional hadoop command arguments to pass. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### stat(paths) Source: https://snakebite.readthedocs.io/en/latest/client.html Retrieves file status information for a given path, including size, owner, permissions, and timestamps. ```APIDOC ## stat(paths) ### Description Returns file status information for the specified path. ### Method stat ### Parameters - **paths** (string) - Required - Path to stat ### Returns A dictionary containing file metadata such as blocksize, owner, length, access_time, group, permission, file_type, path, modification_time, and block_replication. ### Example ``` >>> client.stat(['/index.asciidoc']) {'blocksize': 134217728L, 'owner': u'wouter', 'length': 100L, 'access_time': 1367317326510L, 'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'path': '/index.asciidoc', 'modification_time': 1367317326522L, 'block_replication': 1} ``` ``` -------------------------------- ### du(paths, include_toplevel=False, include_children=True) Source: https://snakebite.readthedocs.io/en/latest/client.html Returns size information for the given paths. The function accepts a list of paths and optional flags to include the top-level path and child nodes. ```APIDOC ## du(paths, include_toplevel=False, include_children=True) ### Description Returns size information for the given paths. ### Method N/A (Python function) ### Endpoint N/A (Python function) ### Parameters - **paths** (list) - Required - Paths to du - **include_toplevel** (boolean) - Optional - Include the given path in the result. If the path is a file, include_toplevel is always True. - **include_children** (boolean) - Optional - Include child nodes in the result. ### Returns A generator that yields dictionaries. ### Example ``` >>> list(client.du(['/'])) [{'path': '/Makefile', 'length': 6783L}, {'path': '/build', 'length': 244778L}, {'path': '/index.asciidoc', 'length': 100L}, {'path': '/source', 'length': 8524L}] ``` ### Example (Directory only) ``` >>> list(client.du(['/'], include_toplevel=True, include_children=False)) [{'path': '/', 'length': 260185L}] ``` ``` -------------------------------- ### terminate Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Terminates the mini cluster explicitly when the program ends. ```APIDOC ## Function: terminate ### Description Terminate the cluster. Must be called explicitly when the program ends since the cluster runs as a subprocess. ### Parameters None. ### Returns No return value. ``` -------------------------------- ### is_zero_bytes_file Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Checks if a file has zero bytes size in the mini cluster. ```APIDOC ## Function: is_zero_bytes_file ### Description Return True if the file at the specified path is zero bytes in size, else return False. ### Parameters - **path** (string) - Required - The path to the file to check. ### Returns A boolean indicating if the file is zero bytes. ``` -------------------------------- ### tail(path, tail_length=1024, append=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Reads the tail end of a file, returning the last portion of the file content. ```APIDOC ## tail(path, tail_length=1024, append=False) ### Description Displays the end of a file, defaulting to the last 1KB. Supports reading up to the Hadoop block size. ### Method tail ### Parameters - **path** (string) - Required - Path to read - **tail_length** (int) - Optional - Number of bytes to read from the end of the file. Default is 1024. - **append** (bool) - Optional - Currently not implemented. Default is False. ### Returns A generator that yields strings. ``` -------------------------------- ### count(paths) Source: https://snakebite.readthedocs.io/en/latest/client.html Counts files and directories in specified HDFS paths. ```APIDOC ## count(paths) ### Description Count files in a path. ### Parameters - **paths** (list) - Required - List of paths to count ### Returns A generator that yields dictionaries. ### Example ``` >>> list(client.count(['/'])) [{'spaceConsumed': 260185L, 'quota': 2147483647L, 'spaceQuota': 18446744073709551615L, 'length': 260185L, 'directoryCount': 9L, 'path': '/', 'fileCount': 34L}] ``` ``` -------------------------------- ### count Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Performs a hadoop count operation on a specified path in the mini cluster. ```APIDOC ## Function: count ### Description Perform `count` on a path in the mini cluster. ### Parameters - **src** (string) - Required - The path on which to perform the count operation. ### Returns A list of maps that are snakebite compatible. ``` -------------------------------- ### is_directory Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Checks if a path is a directory in the mini cluster. ```APIDOC ## Function: is_directory ### Description Return True if the specified path is a directory, False if it is NOT a directory. ### Parameters - **path** (string) - Required - The path to check. ### Returns A boolean indicating if the path is a directory. ``` -------------------------------- ### is_files Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Checks if a path represents a file in the mini cluster. ```APIDOC ## Function: is_files ### Description Return True if the specified path is a file, False if it is NOT a file. ### Parameters - **path** (string) - Required - The path to check. ### Returns A boolean indicating if the path is a file. ``` -------------------------------- ### is_greater_then_zero_bytes Source: https://snakebite.readthedocs.io/en/latest/minicluster.html Checks if a file is larger than zero bytes in the mini cluster. ```APIDOC ## Function: is_greater_then_zero_bytes ### Description Return True if the file at the specified path is greater than zero bytes in size, False otherwise. ### Parameters - **path** (string) - Required - The path to the file to check. ### Returns A boolean indicating if the file is greater than zero bytes. ``` -------------------------------- ### getmerge(path, dst, newline=False, check_crc=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Merges files from a directory into a single local file. The function takes a source directory and a destination path, with an optional newline flag. ```APIDOC ## getmerge(path, dst, newline=False, check_crc=False) ### Description Gets all the files in the directories that match the source file pattern and merges and sorts them to only one file on the local filesystem. ### Method N/A (Python function) ### Endpoint N/A (Python function) ### Parameters - **paths** (string) - Required - Directory containing files that will be merged - **dst** (string) - Required - Path of file that will be written - **nl** (boolean) - Optional - Add a newline character at the end of each file. ### Returns String content of the merged file at dst. ``` -------------------------------- ### chgrp(paths, group, recurse=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Changes the group ownership of specified HDFS paths. ```APIDOC ## chgrp(paths, group, recurse=False) ### Description Change the group of paths. ### Parameters - **paths** (list) - Required - List of paths to chgrp - **group** (string) - Required - New group - **recurse** (boolean) - Optional - Recursive chgrp ### Returns A generator that yields dictionaries. ``` -------------------------------- ### chown(paths, owner, recurse=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Changes the owner (and optionally group) of specified HDFS paths. ```APIDOC ## chown(paths, owner, recurse=False) ### Description Change the owner for paths. The owner can be specified as user or user:group. ### Parameters - **paths** (list) - Required - List of paths to chmod - **owner** (string) - Required - New owner - **recurse** (boolean) - Optional - Recursive chown ### Returns A generator that yields dictionaries. ### Notes This always includes the toplevel when recursing. ``` -------------------------------- ### rename(paths, dst) Source: https://snakebite.readthedocs.io/en/latest/client.html Renames or moves the given source paths to a destination. The function takes a list of source paths and a destination string. ```APIDOC ## rename(paths, dst) ### Description Renames (moves) path(s) to a destination. ### Method N/A (Python function) ### Endpoint N/A (Python function) ### Parameters - **paths** (list) - Required - Source paths - **dst** (string) - Required - Destination ### Returns A generator that yields dictionaries. ``` -------------------------------- ### rmdir(paths) Source: https://snakebite.readthedocs.io/en/latest/client.html Deletes one or more directories. Returns a generator that yields dictionaries representing the result of each deletion. ```APIDOC ## rmdir(paths) ### Description Deletes a directory. The paths parameter is a list of paths to delete. ### Method rmdir ### Parameters - **paths** (list) - Required - Paths to delete ### Returns A generator that yields dictionaries. ``` -------------------------------- ### chmod(paths, mode, recurse=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Changes the permission mode of specified HDFS paths. ```APIDOC ## chmod(paths, mode, recurse=False) ### Description Change the mode for paths. This returns a list of maps containing the result of the operation. ### Parameters - **paths** (list) - Required - List of paths to chmod - **mode** (int) - Required - Octal mode (e.g. 0o755) - **recurse** (boolean) - Optional - Recursive chmod ### Returns A generator that yields dictionaries. ### Notes The top level directory is always included when recurse=True. ``` -------------------------------- ### delete(paths, recurse=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Deletes specified HDFS paths, optionally recursively. ```APIDOC ## delete(paths, recurse=False) ### Description Delete paths. ### Parameters - **paths** (list) - Required - Paths to delete - **recurse** (boolean) - Optional - Recursive delete (use with care!) ### Returns A generator that yields dictionaries. ### Notes Recursive deletion uses the NameNode recursive deletion functionality instead of letting the client recurse. Hadoops client recurses by itself and thus showing all files and directories that are deleted. Snakebite doesn't. ``` -------------------------------- ### rename2(path, dst, overwriteDest=False) Source: https://snakebite.readthedocs.io/en/latest/client.html Renames a file or folder to a new name within the same directory. This operation does not move the file to a different folder. ```APIDOC ## rename2(path, dst, overwriteDest=False) ### Description Renames (but does not move) a file or folder to a destination within the same folder. This operation always succeeds unless an exception is raised, and the returned dictionary does not contain a 'result' key. ### Method rename2 ### Parameters - **path** (string) - Required - Source path - **dst** (string) - Required - Destination path - **overwriteDest** (boolean) - Optional - If True, overwrite the destination if it exists. Default is False. ### Returns A dictionary or None. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.