Snakebite

https://snakebite.readthedocs.io/en/latest/
Admin
Snakebite is a pure Python HDFS client library and CLI that uses protobuf messages over Hadoop RPC...
Tokens:13,970
Snippets:98
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### ls get directory information example

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.ls with include_toplevel=True and include_children=False, returning only the directory itself.

```python
>>> list(client.ls(["/source"], include_toplevel=True, include_children=False))
[{'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}]
```

--------------------------------

### HAClient example: configure namenodes and list root

Source: https://snakebite.readthedocs.io/en/latest/client.html

Sets up a high-availability client with two namenodes and trash enabled. The HAClient is backwards compatible with the vanilla Client and can also be used for non-HA clusters. The example lists the root directory and prints each entry.

```python
>>> from snakebite.client import HAClient
>>> from snakebite.namenode import Namenode
>>> n1 = Namenode("namenode1.mydomain", 8020)
>>> n2 = Namenode("namenode2.mydomain", 8020)
>>> client = HAClient([n1, n2], use_trash=True)
>>> for x in client.ls(['/']):
...     print x

```

--------------------------------

### AutoConfigClient example: list root directory

Source: https://snakebite.readthedocs.io/en/latest/client.html

Creates an AutoConfigClient that reads HADOOP_HOME/conf/hdfs-site.xml and core-site.xml to locate the namenode. The default hadoop_version is 9, which can be overridden via the constructor parameter. The example lists the root directory and prints each entry.

```python
>>> from snakebite.client import AutoConfigClient
>>> client = AutoConfigClient()
>>> for x in client.ls(['/']):
...     print x

```

--------------------------------

### Get server defaults with client.serverdefaults()

Source: https://snakebite.readthedocs.io/en/latest/client.html

Example output of calling client.serverdefaults() without arguments. Returns a dictionary with keys: blockSize, bytesPerChecksum, writePacketSize, replication, fileBufferSize, encryptDataTransfer, trashInterval, checksumType.

```python
>>> client.serverdefaults()
[{'writePacketSize': 65536, 'fileBufferSize': 4096, 'replication': 1, 'bytesPerChecksum': 512, 'trashInterval': 0L, 'blockSize': 134217728L, 'encryptDataTransfer': False, 'checksumType': 2}]

```

--------------------------------

### ls directory listing example

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.ls(["/"]) with default parameters, listing directory contents with file metadata.

```python
>>> list(client.ls(["/"]))
[{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317325431L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/build'}, {'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317326510L, 'block_replication': 1, 'modification_time': 1367317326522L, 'length': 100L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/index.asciidoc'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}]
```

--------------------------------

### ls file listing example

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.ls(["/Makefile"]) for a single file, returning only that file's metadata.

```python
>>> list(client.ls(["/Makefile"]))
[{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}]
```

--------------------------------

### du directory only example

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.du with include_toplevel=True and include_children=False, returning only the top-level path.

```python
>>> list(client.du(['/'], include_toplevel=True, include_children=False))
[{'path': '/', 'length': 260185L}]
```

--------------------------------

### Get filesystem information with df()

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.df() as a dictionary containing used, capacity, under_replicated, missing_blocks, filesystem, remaining, and corrupt_blocks.

```python
>>> client.df()
{'used': 491520L, 'capacity': 120137519104L, 'under_replicated': 0L, 'missing_blocks': 0L, 'filesystem': 'hdfs://localhost:8020', 'remaining': 19669295104L, 'corrupt_blocks': 0L}
```

--------------------------------

### Snakebite configuration file format

Source: https://snakebite.readthedocs.io/en/latest/cli.html

Example of a JSON-formatted configuration file for Snakebite, which can be placed in ~/.snakebiterc or /etc/snakebiterc.

```json
{
    "config_version": 2,
    "skiptrash": true,
    "namenodes": [
        {"host": "mynamenode1", "port": 8020, "version": 9},
        {"host": "mynamenode2", "port": 8020, "version": 9}
    ]
}
```

--------------------------------

### du children example

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.du(['/']) with default parameters, listing child paths and their lengths.

```python
>>> list(client.du(['/']))
[{'path': '/Makefile', 'length': 6783L}, {'path': '/build', 'length': 244778L}, {'path': '/index.asciidoc', 'length': 100L}, {'path': '/source', 'length': 8524L}]
```

--------------------------------

### MiniCluster usage without snakebite.client

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Shows how to instantiate MiniCluster and call ls() without using the snakebite client. The cluster is started at instantiation.

```python
>>> from snakebite.minicluster import MiniCluster
>>> cluster = MiniCluster("/path/to/test/files")
>>> ls_output = cluster.ls(["/"])

```

--------------------------------

### Stat a file with client.stat()

Source: https://snakebite.readthedocs.io/en/latest/client.html

Example output of calling client.stat(['/index.asciidoc']). Returns a dictionary with file metadata such as blocksize, owner, length, access_time, group, permission, file_type, path, modification_time, and block_replication.

```python
>>> client.stat(['/index.asciidoc'])
{'blocksize': 134217728L, 'owner': u'wouter', 'length': 100L, 'access_time': 1367317326510L, 'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'path': '/index.asciidoc', 'modification_time': 1367317326522L, 'block_replication': 1}

```

--------------------------------

### Instantiate Client and list root directory

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows how to create a Client instance and consume the generator returned by ls() to list the root directory. Note that many methods return generators, so they must be consumed to execute.

```python
>>> from snakebite.client import Client
>>> client = Client("localhost", 8020, use_trash=False)
>>> for x in client.ls(['/']):
...     print x
```

--------------------------------

### mkdir

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Creates a directory in the mini cluster, with optional extra arguments.

```APIDOC
## Function: mkdir

### Description
Create a directory in the mini cluster.

### Parameters
- **src** (string) - Required - The path of the directory to create.
- **extra_args** (list) - Optional - Additional hadoop command arguments.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### Client()

Source: https://snakebite.readthedocs.io/en/latest/client.html

Creates a pure Python HDFS client instance with configurable connection and authentication options.

```APIDOC
## Client()

### Description
Creates a pure Python HDFS client instance.

### Constructor
`Client(host, port=8020, hadoop_version=9, use_trash=False, effective_user=None, use_sasl=False, hdfs_namenode_principal=None, sock_connect_timeout=10000, sock_request_timeout=10000, use_datanode_hostname=False)`

### Parameters
- **host** (string) - Required - Hostname or IP address of the NameNode
- **port** (int) - Optional - RPC Port of the NameNode (default: 8020)
- **hadoop_version** (int) - Optional - What hadoop protocol version should be used (default: 9)
- **use_trash** (boolean) - Optional - Use a trash when removing files.
- **effective_user** (string) - Optional - Effective user for the HDFS operations (default: None - current user)
- **use_sasl** (boolean) - Optional - Use SASL authentication or not
- **hdfs_namenode_principal** (string) - Optional - Kerberos principal to use for HDFS
- **sock_connect_timeout** (int) - Optional - Socket connection timeout in seconds
- **sock_request_timeout** (int) - Optional - Request timeout in seconds
- **use_datanode_hostname** (boolean) - Optional - Use hostname instead of IP address to communicate with datanodes

### Example
```
>>> from snakebite.client import Client
>>> client = Client("localhost", 8020, use_trash=False)
```
```

--------------------------------

### ls

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Lists files in a directory of the mini cluster, supporting extra arguments.

```APIDOC
## Function: ls

### Description
List files in a directory in the mini cluster.

### Parameters
- **src** (string) - Required - The source path to list files from.
- **extra_args** (list) - Optional - Additional hadoop command arguments.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### touchz(paths, replication=None, blocksize=None)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Creates a zero-length file or updates the timestamp on an existing zero-length file.

```APIDOC
## touchz(paths, replication=None, blocksize=None)

### Description
Creates a zero-length file or updates the timestamp on an existing zero-length file.

### Method
touchz

### Parameters
- **paths** (list) - Required - Paths to create or update
- **replication** (int) - Optional - Replication factor for the new file.
- **blocksize** (int) - Optional - Block size (in bytes) for the new file.

### Returns
A generator that yields dictionaries.
```

--------------------------------

### mkdir(paths, create_parent=False, mode=493)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Creates directories at the specified paths. The function accepts a list of paths and optional flags for creating parent directories and setting the mode.

```APIDOC
## mkdir(paths, create_parent=False, mode=493)

### Description
Creates a directory.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
- **paths** (list of strings) - Required - Paths to create
- **create_parent** (boolean) - Optional - Also create the parent directories
- **mode** (int) - Optional - Mode the directory should be created with

### Returns
A generator that yields dictionaries.
```

--------------------------------

### Count files in a path

Source: https://snakebite.readthedocs.io/en/latest/client.html

Shows the output of client.count(['/']) as a list of dictionaries containing spaceConsumed, quota, spaceQuota, length, directoryCount, path, and fileCount.

```python
>>> list(client.count(['/']))
[{'spaceConsumed': 260185L, 'quota': 2147483647L, 'spaceQuota': 18446744073709551615L, 'length': 260185L, 'directoryCount': 9L, 'path': '/', 'fileCount': 34L}]
```

--------------------------------

### put

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Uploads a file to the mini cluster from the test files path provided during construction.

```APIDOC
## Function: put

### Description
Uploads a file to HDFS. Takes a source path and a destination path, with the file sourced from `testfiles_path` provided in the constructor.

### Parameters
- **src** (string) - Required - The path to the local file to upload.
- **dst** (string) - Required - The destination path in HDFS.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### serverdefaults(force_reload=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Retrieves the default configuration values from the HDFS server, with optional caching and forced reload.

```APIDOC
## serverdefaults(force_reload=False)

### Description
Gets the HDFS server defaults, caching the results. If force_reload is True or no cached results exist, it queries the server for default parameter values.

### Method
serverdefaults

### Parameters
- **force_reload** (bool) - Optional - If True, reload the server defaults even if they are already cached. Default is False.

### Returns
A dictionary with keys: blockSize, bytesPerChecksum, writePacketSize, replication, fileBufferSize, encryptDataTransfer, trashInterval, checksumType.

### Example
```
>>> client.serverdefaults()
[{'writePacketSize': 65536, 'fileBufferSize': 4096, 'replication': 1, 'bytesPerChecksum': 512, 'trashInterval': 0L, 'blockSize': 134217728L, 'encryptDataTransfer': False, 'checksumType': 2}]
```
```

--------------------------------

### df()

Source: https://snakebite.readthedocs.io/en/latest/client.html

Retrieves filesystem information from the HDFS cluster.

```APIDOC
## df()

### Description
Get FS information.

### Returns
A dictionary.

### Example
```
>>> client.df()
{'used': 491520L, 'capacity': 120137519104L, 'under_replicated': 0L, 'missing_blocks': 0L, 'filesystem': 'hdfs://localhost:8020', 'remaining': 19669295104L, 'corrupt_blocks': 0L}
```
```

--------------------------------

### df

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Performs the hadoop df command on a specified path in the mini cluster.

```APIDOC
## Function: df

### Description
Perform `df` on a path in the mini cluster to display disk usage.

### Parameters
- **src** (string) - Required - The path on which to perform the df operation.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### test(path, exists=False, directory=False, zero_length=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Checks various properties of a path, such as existence, directory status, and zero length.

```APIDOC
## test(path, exists=False, directory=False, zero_length=False)

### Description
Tests whether a path exists, is a directory, or has zero length. The directory and zero_length checks are AND'd together.

### Method
test

### Parameters
- **path** (string) - Required - Path to test
- **exists** (boolean) - Optional - Check if the path exists. Default is False.
- **directory** (boolean) - Optional - Check if the path is a directory. Default is False.
- **zero_length** (boolean) - Optional - Check if the path is zero-length. Default is False.

### Returns
A boolean indicating whether the specified conditions are met.
```

--------------------------------

### ls(paths, recurse=False, include_toplevel=False, include_children=True)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Lists file information for the given paths. The function returns a generator of dictionaries containing file metadata.

```APIDOC
## ls(paths, recurse=False, include_toplevel=False, include_children=True)

### Description
Issues 'ls' command and returns a list of maps that contain file information.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
- **paths** (list) - Required - Paths to list
- **recurse** (boolean) - Optional - Recursive listing
- **include_toplevel** (boolean) - Optional - Include the given path in the listing. If the path is a file, include_toplevel is always True.
- **include_children** (boolean) - Optional - Include child nodes in the listing.

### Returns
A generator that yields dictionaries.

### Example (Directory listing)
```
>>> list(client.ls(["/"]))
[{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317325431L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/build'}, {'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317326510L, 'block_replication': 1, 'modification_time': 1367317326522L, 'length': 100L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/index.asciidoc'}, {'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}]
```

### Example (File listing)
```
>>> list(client.ls(["/Makefile"]))
[{'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'access_time': 1367317324982L, 'block_replication': 1, 'modification_time': 1367317325346L, 'length': 6783L, 'blocksize': 134217728L, 'owner': u'wouter', 'path': '/Makefile'}]
```

### Example (Get directory information)
```
>>> list(client.ls(["/source"], include_toplevel=True, include_children=False))
[{'group': u'supergroup', 'permission': 493, 'file_type': 'd', 'access_time': 0L, 'block_replication': 0, 'modification_time': 1367317326628L, 'length': 0L, 'blocksize': 0L, 'owner': u'wouter', 'path': '/source'}]
```
```

--------------------------------

### cat(paths, check_crc=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Fetches file contents from HDFS and yields them as strings.

```APIDOC
## cat(paths, check_crc=False)

### Description
Fetch all files that match the source file pattern and display their content on stdout.

### Parameters
- **paths** (list of strings) - Required - Paths to display
- **check_crc** (boolean) - Optional - Check for checksum errors

### Returns
A generator that yields strings.
```

--------------------------------

### exists

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Checks whether a given path exists in the mini cluster.

```APIDOC
## Function: exists

### Description
Return True if the specified path exists, False if it doesn't.

### Parameters
- **path** (string) - Required - The path to check for existence.

### Returns
A boolean value indicating whether the path exists.
```

--------------------------------

### Snakebite CLI usage and command reference

Source: https://snakebite.readthedocs.io/en/latest/cli.html

General syntax for the snakebite command, including available options and a list of supported HDFS commands.

```bash
snakebite [general options] cmd [arguments]
general options:
  -D --debug                     Show debug information
  -V --version                   Hadoop protocol version (default:9)
  -h --help                      show help
  -j --json                      JSON output
  -n --namenode                  namenode host
  -p --port                      namenode RPC port (default: 8020)
  -v --ver                       Display snakebite version

commands:
  cat [paths]                    copy source paths to stdout
  chgrp <grp> [paths]            change group
  chmod <mode> [paths]           change file mode (octal)
  chown <owner:grp> [paths]      change owner
  copyToLocal [paths] dst        copy paths to local file system destination
  count [paths]                  display stats for paths
  df                             display fs stats
  du [paths]                     display disk usage statistics
  get file dst                   copy files to local file system destination
  getmerge dir dst               concatenates files in source dir into destination local file
  ls [paths]                     list a path
  mkdir [paths]                  create directories
  mkdirp [paths]                 create directories and their parents
  mv [paths] dst                 move paths to destination
  rm [paths]                     remove paths
  rmdir [dirs]                   delete a directory
  serverdefaults                 show server information
  setrep <rep> [paths]           set replication factor
  stat [paths]                   stat information
  tail path                      display last kilobyte of the file to stdout
  test path                      test a path
  text path [paths]              output file in text format
  touchz [paths]                 creates a file of zero length
  usage <cmd>                    show cmd usage

to see command-specific options use: snakebite [cmd] --help
```

--------------------------------

### copyToLocal(paths, dst, check_crc=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Copies files from HDFS to the local filesystem.

```APIDOC
## copyToLocal(paths, dst, check_crc=False)

### Description
Copy files that match the file source pattern to the local name. Source is kept. When copying multiple files, the destination must be a directory.

### Parameters
- **paths** (list of strings) - Required - Paths to copy
- **dst** (string) - Required - Destination path
- **check_crc** (boolean) - Optional - Check for checksum errors

### Returns
A generator that yields strings.
```

--------------------------------

### MiniCluster usage with snakebite.client

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Shows how to use MiniCluster with the snakebite Client. The client connects to localhost on the cluster's port, and then ls() is called on the client.

```python
>>> from snakebite.minicluster import MiniCluster
>>> from snakebite.client import Client
>>> cluster = MiniCluster("/path/to/test/files")
>>> client = Client('localhost', cluster.port)
>>> ls_output = client.ls(["/"])

```

--------------------------------

### setrep(paths, replication, recurse=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Sets the replication factor for one or more paths. Optionally applies the setting recursively.

```APIDOC
## setrep(paths, replication, recurse=False)

### Description
Sets the replication factor for the specified paths. Optionally applies the replication factor recursively to subdirectories.

### Method
setrep

### Parameters
- **paths** (list) - Required - Paths to set replication factor for
- **replication** (int) - Required - Replication factor
- **recurse** (boolean) - Optional - If True, apply replication factor recursively. Default is False.

### Returns
A generator that yields dictionaries.
```

--------------------------------

### text(paths, check_crc=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Displays the contents of a file in text format, supporting gzip and bzip2 compressed files.

```APIDOC
## text(paths, check_crc=False)

### Description
Outputs the contents of a file in text format. Supported formats are gzip and bzip2.

### Method
text

### Parameters
- **paths** (list of strings) - Required - Paths to display
- **check_crc** (boolean) - Optional - If True, check for checksum errors. Default is False.

### Returns
A generator that yields strings.
```

--------------------------------

### du

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Performs the du command on a path in the mini cluster, allowing extra hadoop arguments.

```APIDOC
## Function: du

### Description
Perform `du` on a path in the mini cluster to show disk usage details.

### Parameters
- **src** (string) - Required - The path on which to perform the du operation.
- **extra_args** (list) - Optional - Additional hadoop command arguments to pass.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### stat(paths)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Retrieves file status information for a given path, including size, owner, permissions, and timestamps.

```APIDOC
## stat(paths)

### Description
Returns file status information for the specified path.

### Method
stat

### Parameters
- **paths** (string) - Required - Path to stat

### Returns
A dictionary containing file metadata such as blocksize, owner, length, access_time, group, permission, file_type, path, modification_time, and block_replication.

### Example
```
>>> client.stat(['/index.asciidoc'])
{'blocksize': 134217728L, 'owner': u'wouter', 'length': 100L, 'access_time': 1367317326510L, 'group': u'supergroup', 'permission': 420, 'file_type': 'f', 'path': '/index.asciidoc', 'modification_time': 1367317326522L, 'block_replication': 1}
```
```

--------------------------------

### du(paths, include_toplevel=False, include_children=True)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Returns size information for the given paths. The function accepts a list of paths and optional flags to include the top-level path and child nodes.

```APIDOC
## du(paths, include_toplevel=False, include_children=True)

### Description
Returns size information for the given paths.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
- **paths** (list) - Required - Paths to du
- **include_toplevel** (boolean) - Optional - Include the given path in the result. If the path is a file, include_toplevel is always True.
- **include_children** (boolean) - Optional - Include child nodes in the result.

### Returns
A generator that yields dictionaries.

### Example
```
>>> list(client.du(['/']))
[{'path': '/Makefile', 'length': 6783L}, {'path': '/build', 'length': 244778L}, {'path': '/index.asciidoc', 'length': 100L}, {'path': '/source', 'length': 8524L}]
```

### Example (Directory only)
```
>>> list(client.du(['/'], include_toplevel=True, include_children=False))
[{'path': '/', 'length': 260185L}]
```
```

--------------------------------

### terminate

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Terminates the mini cluster explicitly when the program ends.

```APIDOC
## Function: terminate

### Description
Terminate the cluster. Must be called explicitly when the program ends since the cluster runs as a subprocess.

### Parameters
None.

### Returns
No return value.
```

--------------------------------

### is_zero_bytes_file

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Checks if a file has zero bytes size in the mini cluster.

```APIDOC
## Function: is_zero_bytes_file

### Description
Return True if the file at the specified path is zero bytes in size, else return False.

### Parameters
- **path** (string) - Required - The path to the file to check.

### Returns
A boolean indicating if the file is zero bytes.
```

--------------------------------

### tail(path, tail_length=1024, append=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Reads the tail end of a file, returning the last portion of the file content.

```APIDOC
## tail(path, tail_length=1024, append=False)

### Description
Displays the end of a file, defaulting to the last 1KB. Supports reading up to the Hadoop block size.

### Method
tail

### Parameters
- **path** (string) - Required - Path to read
- **tail_length** (int) - Optional - Number of bytes to read from the end of the file. Default is 1024.
- **append** (bool) - Optional - Currently not implemented. Default is False.

### Returns
A generator that yields strings.
```

--------------------------------

### count(paths)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Counts files and directories in specified HDFS paths.

```APIDOC
## count(paths)

### Description
Count files in a path.

### Parameters
- **paths** (list) - Required - List of paths to count

### Returns
A generator that yields dictionaries.

### Example
```
>>> list(client.count(['/']))
[{'spaceConsumed': 260185L, 'quota': 2147483647L, 'spaceQuota': 18446744073709551615L, 'length': 260185L, 'directoryCount': 9L, 'path': '/', 'fileCount': 34L}]
```
```

--------------------------------

### count

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Performs a hadoop count operation on a specified path in the mini cluster.

```APIDOC
## Function: count

### Description
Perform `count` on a path in the mini cluster.

### Parameters
- **src** (string) - Required - The path on which to perform the count operation.

### Returns
A list of maps that are snakebite compatible.
```

--------------------------------

### is_directory

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Checks if a path is a directory in the mini cluster.

```APIDOC
## Function: is_directory

### Description
Return True if the specified path is a directory, False if it is NOT a directory.

### Parameters
- **path** (string) - Required - The path to check.

### Returns
A boolean indicating if the path is a directory.
```

--------------------------------

### is_files

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Checks if a path represents a file in the mini cluster.

```APIDOC
## Function: is_files

### Description
Return True if the specified path is a file, False if it is NOT a file.

### Parameters
- **path** (string) - Required - The path to check.

### Returns
A boolean indicating if the path is a file.
```

--------------------------------

### is_greater_then_zero_bytes

Source: https://snakebite.readthedocs.io/en/latest/minicluster.html

Checks if a file is larger than zero bytes in the mini cluster.

```APIDOC
## Function: is_greater_then_zero_bytes

### Description
Return True if the file at the specified path is greater than zero bytes in size, False otherwise.

### Parameters
- **path** (string) - Required - The path to the file to check.

### Returns
A boolean indicating if the file is greater than zero bytes.
```

--------------------------------

### getmerge(path, dst, newline=False, check_crc=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Merges files from a directory into a single local file. The function takes a source directory and a destination path, with an optional newline flag.

```APIDOC
## getmerge(path, dst, newline=False, check_crc=False)

### Description
Gets all the files in the directories that match the source file pattern and merges and sorts them to only one file on the local filesystem.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
- **paths** (string) - Required - Directory containing files that will be merged
- **dst** (string) - Required - Path of file that will be written
- **nl** (boolean) - Optional - Add a newline character at the end of each file.

### Returns
String content of the merged file at dst.
```

--------------------------------

### chgrp(paths, group, recurse=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Changes the group ownership of specified HDFS paths.

```APIDOC
## chgrp(paths, group, recurse=False)

### Description
Change the group of paths.

### Parameters
- **paths** (list) - Required - List of paths to chgrp
- **group** (string) - Required - New group
- **recurse** (boolean) - Optional - Recursive chgrp

### Returns
A generator that yields dictionaries.
```

--------------------------------

### chown(paths, owner, recurse=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Changes the owner (and optionally group) of specified HDFS paths.

```APIDOC
## chown(paths, owner, recurse=False)

### Description
Change the owner for paths. The owner can be specified as user or user:group.

### Parameters
- **paths** (list) - Required - List of paths to chmod
- **owner** (string) - Required - New owner
- **recurse** (boolean) - Optional - Recursive chown

### Returns
A generator that yields dictionaries.

### Notes
This always includes the toplevel when recursing.
```

--------------------------------

### rename(paths, dst)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Renames or moves the given source paths to a destination. The function takes a list of source paths and a destination string.

```APIDOC
## rename(paths, dst)

### Description
Renames (moves) path(s) to a destination.

### Method
N/A (Python function)

### Endpoint
N/A (Python function)

### Parameters
- **paths** (list) - Required - Source paths
- **dst** (string) - Required - Destination

### Returns
A generator that yields dictionaries.
```

--------------------------------

### rmdir(paths)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Deletes one or more directories. Returns a generator that yields dictionaries representing the result of each deletion.

```APIDOC
## rmdir(paths)

### Description
Deletes a directory. The paths parameter is a list of paths to delete.

### Method
rmdir

### Parameters
- **paths** (list) - Required - Paths to delete

### Returns
A generator that yields dictionaries.
```

--------------------------------

### chmod(paths, mode, recurse=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Changes the permission mode of specified HDFS paths.

```APIDOC
## chmod(paths, mode, recurse=False)

### Description
Change the mode for paths. This returns a list of maps containing the result of the operation.

### Parameters
- **paths** (list) - Required - List of paths to chmod
- **mode** (int) - Required - Octal mode (e.g. 0o755)
- **recurse** (boolean) - Optional - Recursive chmod

### Returns
A generator that yields dictionaries.

### Notes
The top level directory is always included when recurse=True.
```

--------------------------------

### delete(paths, recurse=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Deletes specified HDFS paths, optionally recursively.

```APIDOC
## delete(paths, recurse=False)

### Description
Delete paths.

### Parameters
- **paths** (list) - Required - Paths to delete
- **recurse** (boolean) - Optional - Recursive delete (use with care!)

### Returns
A generator that yields dictionaries.

### Notes
Recursive deletion uses the NameNode recursive deletion functionality instead of letting the client recurse. Hadoops client recurses by itself and thus showing all files and directories that are deleted. Snakebite doesn't.
```

--------------------------------

### rename2(path, dst, overwriteDest=False)

Source: https://snakebite.readthedocs.io/en/latest/client.html

Renames a file or folder to a new name within the same directory. This operation does not move the file to a different folder.

```APIDOC
## rename2(path, dst, overwriteDest=False)

### Description
Renames (but does not move) a file or folder to a destination within the same folder. This operation always succeeds unless an exception is raised, and the returned dictionary does not contain a 'result' key.

### Method
rename2

### Parameters
- **path** (string) - Required - Source path
- **dst** (string) - Required - Destination path
- **overwriteDest** (boolean) - Optional - If True, overwrite the destination if it exists. Default is False.

### Returns
A dictionary or None.
```
Tokens:
Raw