PyHDFS

https://pyhdfs.readthedocs.io/en/latest/
Admin
PyHDFS is a WebHDFS client library with support for NameNode high availability and automatic error...
Tokens:19,770
Snippets:135
Trust Score:9.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### list_xattrs

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Get all of the xattr names for a file or directory.

```APIDOC
## list_xattrs

### Description
Get all of the xattr names for a file or directory.

### Method
list_xattrs

### Parameters
#### Path Parameters
- **_path** (str) - The path to get xattr names from.

#### Other Parameters
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
list
```

--------------------------------

### Get Home Directory - Pyhdfs

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Returns the current user's home directory in the HDFS filesystem. Asserts that the response is a string.

```python
def get_home_directory(self, **kwargs: _PossibleArgumentTypes) -> str:
        """Return the current user's home directory in this filesystem."""
        response = _json(self._get("/", "GETHOMEDIRECTORY", **kwargs))["Path"]
        assert isinstance(response, str), type(response)
        return response
```

--------------------------------

### HTTP Request Methods in Pyhdfs

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

These methods abstract common HTTP request types (GET, PUT, POST, DELETE) used for interacting with HDFS. They handle path construction, operation specification, and expected status codes.

```python
    def _get(
        self,
        path: str,
        op: str,
        expected_status: Any = HTTPStatus.OK,
        **kwargs: _PossibleArgumentTypes,
    ) -> requests.Response:
        return self._request("get", path, op, expected_status, **kwargs)

```

```python
    def _put(
        self,
        path: str,
        op: str,
        expected_status: Any = HTTPStatus.OK,
        **kwargs: _PossibleArgumentTypes,
    ) -> requests.Response:
        return self._request("put", path, op, expected_status, **kwargs)

```

```python
    def _post(
        self,
        path: str,
        op: str,
        expected_status: Any = HTTPStatus.OK,
        **kwargs: _PossibleArgumentTypes,
    ) -> requests.Response:
        return self._request("post", path, op, expected_status, **kwargs)

```

```python
    def _delete(
        self,
        path: str,
        op: str,
        expected_status: Any = HTTPStatus.OK,
        **kwargs: _PossibleArgumentTypes,
    ) -> requests.Response:
        return self._request("delete", path, op, expected_status, **kwargs)

```

--------------------------------

### Initialize HDFS Client

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Create a new HDFS client instance. Configure hosts, user, timeouts, and retry logic. Ensure max_tries is at least 1 and retry_delay is non-negative. Cannot override specific requests arguments.

```python
def __init__(
        self,
        hosts: str | Iterable[str] = "localhost",
        randomize_hosts: bool = True,
        user_name: str | None = None,
        timeout: float = 20,
        max_tries: int = 2,
        retry_delay: float = 5,
        scheme: str = "http",
        requests_session: requests.Session | None = None,
        requests_kwargs: dict[str, Any] | None = None,
    ) -> None:
        """Create a new HDFS client"""
        if max_tries < 1:
            raise ValueError(f"Invalid max_tries: {max_tries}")
        if retry_delay < 0:
            raise ValueError(f"Invalid retry_delay: {retry_delay}")
        self.randomize_hosts = randomize_hosts
        self.hosts = self._parse_hosts(hosts)
        if not self.hosts:
            raise ValueError("No hosts given")
        if randomize_hosts:
            self.hosts = list(self.hosts)
            random.shuffle(self.hosts)
        self.timeout = timeout
        self.max_tries = max_tries
        self.retry_delay = retry_delay
        self.scheme = scheme.lower()
        self.user_name = user_name or os.environ.get(
            "HADOOP_USER_NAME", getpass.getuser()
        )
        self._last_time_recorded_active: float | None = None
        self._requests_session = requests_session or cast(
            requests.Session, requests.api
        )
        self._requests_kwargs = requests_kwargs or {}
        for k in ("method", "url", "data", "timeout", "stream", "params"):
            if k in self._requests_kwargs:
                raise ValueError(f"Cannot override requests argument {k}")
```

--------------------------------

### Get File Checksum - Pyhdfs

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Retrieves the checksum of a file in HDFS. This operation involves an initial redirect and a subsequent GET request to fetch the checksum data.

```python
def get_file_checksum(
        self,
        path: str,
        **kwargs: _PossibleArgumentTypes
    ) -> FileChecksum:
        """Get the checksum of a file.

        :rtype: :py:class:`FileChecksum`
        """
        metadata_response = self._get(
            path,
            "GETFILECHECKSUM",
            expected_status=HTTPStatus.TEMPORARY_REDIRECT,
            **kwargs,
        )
        assert not metadata_response.content
        data_response = self._requests_session.get(
            metadata_response.headers["location"],
            **self._requests_kwargs
        )
        _check_response(data_response)
        return FileChecksum(**_json(data_response)["FileChecksum"])
```

--------------------------------

### HDFS Client Initialization

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

This section details the constructor for the HDFS client, outlining the parameters used to configure the connection and behavior of the client.

```APIDOC
## HDFS Client Initialization

### Description
Creates a new HDFS client instance with configurable connection parameters and retry logic.

### Method
`__init__`

### Parameters
- **hosts** (str | Iterable[str]) - Optional - Comma or semicolon-separated list of HDFS NameNode hosts. Defaults to 'localhost'.
- **randomize_hosts** (bool) - Optional - Whether to randomize the order of hosts for connection attempts. Defaults to True.
- **user_name** (str | None) - Optional - The Hadoop user to run as. Defaults to HADOOP_USER_NAME environment variable or the current system user.
- **timeout** (float) - Optional - How long to wait on a single NameNode in seconds before moving on. Defaults to 20.
- **max_tries** (int) - Optional - How many times to retry a request for each NameNode. Defaults to 2.
- **retry_delay** (float) - Optional - How long to wait in seconds before going through NameNodes again. Defaults to 5.
- **scheme** (str) - Optional - The scheme to use for connections ('http' or 'https'). Defaults to 'http'.
- **requests_session** (requests.Session | None) - Optional - A requests.Session object for advanced usage. If absent, a new session is created per request.
- **requests_kwargs** (dict[str, Any] | None) - Optional - Additional keyword arguments to pass to the requests library.
```

--------------------------------

### walk

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

See `os.walk` for documentation.

```APIDOC
## walk

### Description
See `os.walk` for documentation.

### Method
walk

### Parameters
#### Path Parameters
- **_top** (str) - The starting directory for the walk.
- **topdown** (bool) - If true, directories are visited top-down. If false, directories are visited bottom-up. Defaults to True.
- **onerror** (Callable[[HdfsException], None] | None) - A function to call if an error occurs during the walk. Defaults to None.

#### Other Parameters
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
Iterator[tuple[str, list[str], list[str]]]
```

--------------------------------

### Get Extended Attributes - Pyhdfs

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Retrieves one or more extended attribute (xattr) values for a file or directory. Supports specifying a single xattr name, a list of names, or None to get all attributes. Allows specifying encoding (text, hex, base64).

```python
def get_xattrs(
        self,
        path: str,
        xattr_name: str | list[str] | None = None,
        encoding: str = "text",
        **kwargs: _PossibleArgumentTypes,
    ) -> dict[str, bytes | str | None]:
        """Get one or more xattr values for a file or directory.

        :param xattr_name: ``str`` to get one attribute, ``list`` to get multiple attributes,
            ``None`` to get all attributes.
        :param encoding: ``text`` | ``hex`` | ``base64``, defaults to ``text``

        :returns: Dictionary mapping xattr name to value. With text encoding, the value will be a
            unicode string. With hex or base64 encoding, the value will be a byte array.
        :rtype: dict
        """
        kwargs["xattr.name"] = xattr_name
```

--------------------------------

### mkdirs

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Create a directory with the provided permission.

```APIDOC
## mkdirs

### Description
Create a directory with the provided permission. The permission of the directory is set to be the provided permission as in setPermission, not permission&~umask.

### Method
mkdirs

### Parameters
#### Path Parameters
- **_path** (str) - The path for the directory to be created.

#### Other Parameters
- **permission** (octal) - The permission of a file/directory. Any radix-8 integer (leading zeros may be omitted.).
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Returns
true if the directory creation succeeds; false otherwise

### Return Type
bool
```

--------------------------------

### Get Active NameNode Address

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Retrieves the address of the currently active NameNode. It includes a caching mechanism to avoid frequent lookups, controlled by the `max_staleness` parameter. Raises `HdfsNoServerException` if no active NameNode can be found.

```python
def get_active_namenode(self, max_staleness: float | None = None) -> str:
        """Return the address of the currently active NameNode.

        :param max_staleness: This function caches the active NameNode. If this age of this cached
            result is less than ``max_staleness`` seconds, return it. Otherwise, or if this
            parameter is None, do a lookup.
        :type max_staleness: float
        :raises HdfsNoServerException: can't find an active NameNode
        """
        if (
            max_staleness is None
            or self._last_time_recorded_active is None
            or self._last_time_recorded_active < time.time() - max_staleness
        ):
            # Make a cheap request and rely on the reordering in self._record_last_active
            self.get_file_status("/")
        return self.hosts[0]
```

--------------------------------

### create

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Creates a file at the specified path with the provided data. Supports options for overwriting, block size, replication, permissions, and buffer size.

```APIDOC
## create

### Description
Create a file at the given path.

### Parameters
#### Path Parameters
- **path** (str) - The path to create the file at.
- **data** (bytes | IO[bytes]) - The data to write to the file, can be bytes or a file-like object.

#### Keyword Arguments
- **overwrite** (bool) - If a file already exists, should it be overwritten?
- **blocksize** (long) - The block size of a file.
- **replication** (short) - The number of replications of a file.
- **permission** (octal) - The permission of a file/directory. Any radix-8 integer (leading zeros may be omitted.)
- **buffersize** (int) - The size of the buffer used in transferring data.
```

--------------------------------

### HdfsClient Initialization

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Initializes the HdfsClient with a list of NameNode hosts. Supports randomization and optional port specification.

```APIDOC
## HdfsClient

### Description
HDFS client backed by WebHDFS. All functions take arbitrary query parameters to pass to WebHDFS, in addition to any documented keyword arguments. If multiple HA NameNodes are given, all functions submit HTTP requests to both NameNodes until they find the active NameNode.

### Parameters
#### Path Parameters
- **hosts** (list or str) - Required - List of NameNode HTTP host:port strings, either as ``list`` or a comma separated string. Port defaults to 50070 if left unspecified.
- **randomize_hosts** (bool) - Optional - By default randomize host selection.
```

--------------------------------

### Create HDFS Snapshot

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Creates a snapshot of a directory in HDFS. Returns the path of the created snapshot.

```python
response = _json(self._put(path, "CREATESNAPSHOT", **kwargs))["Path"]
        assert isinstance(response, str), type(response)
        return response
```

--------------------------------

### create_symlink

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Create a symbolic link at ``link`` pointing to ``destination``.

```APIDOC
## create_symlink

### Description
Create a symbolic link at ``link`` pointing to ``destination``.

### Method
PUT

### Endpoint
/{link}?op=CREATESYMLINK

### Parameters
#### Path Parameters
- **link** (str) - Required - The path to be created that points to target.

#### Query Parameters
- **destination** (str) - Required - The target of the symbolic link.
- **createParent** (bool) - Optional - If the parent directories do not exist, should they be created?

### Response
#### Success Response (200)
- **None** - Indicates successful creation.
```

--------------------------------

### open

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Return a file-like object for reading the given HDFS path.

```APIDOC
## open

### Description
Return a file-like object for reading the given HDFS path.

### Method
open

### Parameters
#### Path Parameters
- **_path** (str) - The HDFS path to open.

#### Other Parameters
- **offset** (long) - The starting byte position.
- **length** (long) - The number of bytes to be processed.
- **buffersize** (int) - The size of the buffer used in transferring data.
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
file-like object
```

--------------------------------

### create_symlink

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Creates a symbolic link at a specified path pointing to a destination. Optionally creates parent directories if they do not exist.

```APIDOC
## create_symlink

### Description
Create a symbolic link at `link` pointing to `destination`.

### Parameters
* **link** (str) - The path for the symbolic link to be created.
* **destination** (str) - The target path the symbolic link will point to.
* **createParent** (bool) - If true, creates parent directories if they do not exist.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.

### Raises
* **HdfsUnsupportedOperationException** - This feature doesn’t actually work, at least on CDH 5.3.0.
```

--------------------------------

### create

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Creates a new file at the specified path in HDFS. Supports uploading data from bytes or a file-like object.

```APIDOC
## create

### Description
Create a file at the given path.

### Parameters
* **_path** (str) - The path where the file will be created.
* **_data** (IO[bytes] | bytes) - The data to write to the file, can be bytes or a file-like object.
* **overwrite** (bool) - If true, overwrites the file if it already exists.
* **blocksize** (long) - The block size for the file.
* **replication** (short) - The number of replications for the file.
* **permission** (octal) - The permission of the file/directory (e.g., '0755').
* **buffersize** (int) - The size of the buffer used in transferring data.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.
```

--------------------------------

### walk

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Traverse a directory tree, yielding file and directory names.

```APIDOC
## walk

### Description
Traverse a directory tree, yielding file and directory names. This method mimics the behavior of `os.walk`.

### Method
GET (assumed based on internal list_status calls)

### Endpoint
/{top}?op=LISTSTATUS (and subsequent calls for subdirectories)

### Parameters
#### Path Parameters
- **top** (str) - The starting directory for the traversal.

#### Query Parameters
- **topdown** (bool, optional) - If True, directories are yielded before their subdirectories. Defaults to True.
- **onerror** (Callable[[HdfsException], None] | None, optional) - A function to call if an error occurs during traversal.
- **kwargs** (_PossibleArgumentTypes) - Additional keyword arguments.

### Yields
- **tuple[str, list[str], list[str]]** - A tuple containing the current directory path, a list of subdirectory names, and a list of file names.

### Throws
- **HdfsException** - If an error occurs during directory listing and `onerror` is not provided.
```

--------------------------------

### create_snapshot

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Creates a snapshot of a directory in HDFS. Returns the path of the created snapshot.

```APIDOC
## create_snapshot

### Description
Create a snapshot.

### Parameters
* **path** (str) - The directory where snapshots will be taken.
* **snapshotname** (str) - The name of the snapshot.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.

### Returns
* **str** - The path of the created snapshot.
```

--------------------------------

### HdfsClient Methods

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

This section details the various methods available on the HdfsClient class for interacting with the Hadoop Distributed File System.

```APIDOC
## HdfsClient Methods

### Description
Provides a comprehensive set of methods for file and directory operations in HDFS, including creation, deletion, copying, and status retrieval.

### Methods
- `append(hdfs_path, content, overwrite=False)`: Appends content to an existing file.
- `concat(dst_path, srcs_paths)`: Concatenates multiple source files into a destination file.
- `copy_from_local(local_path, hdfs_path)`: Copies a file from the local filesystem to HDFS.
- `copy_to_local(hdfs_path, local_path)`: Copies a file from HDFS to the local filesystem.
- `create(hdfs_path, content='', overwrite=False)`: Creates a new file in HDFS with optional content.
- `create_snapshot(snapshot_path, path)`: Creates a snapshot of a directory.
- `create_symlink(target_path, link_path)`: Creates a symbolic link.
- `delete(hdfs_path, recursive=False)`: Deletes a file or directory.
- `delete_snapshot(snapshot_path)`: Deletes a snapshot.
- `exists(hdfs_path)`: Checks if a path exists in HDFS.
- `get_active_namenode()`: Retrieves the active NameNode.
- `get_content_summary(hdfs_path)`: Gets the content summary of a file or directory.
- `get_file_checksum(hdfs_path)`: Gets the checksum of a file.
- `get_file_status(hdfs_path)`: Gets the status of a file or directory.
- `get_home_directory()`: Gets the current user's home directory.
- `get_xattrs(hdfs_path)`: Gets extended attributes of a file or directory.
- `list_status(hdfs_path)`: Lists the status of files and directories within a given path.
- `list_xattrs(hdfs_path)`: Lists extended attributes of a file or directory.
- `listdir(hdfs_path)`: Lists the contents of a directory.
- `mkdirs(hdfs_path)`: Creates directories in HDFS.
- `open(hdfs_path)`: Opens a file for reading.
- `remove_xattr(hdfs_path, xattr_name)`: Removes an extended attribute.
- `rename(old_path, new_path)`: Renames a file or directory.
- `rename_snapshot(snapshot_path, rename_path)`: Renames a snapshot.
- `set_owner(hdfs_path, owner=None, group=None)`: Sets the owner and group of a file or directory.
- `set_permission(hdfs_path, permission)`: Sets the permission of a file or directory.
- `set_replication(hdfs_path, replication)`: Sets the replication factor of a file.
- `set_times(hdfs_path, atime, mtime)`: Sets the access and modification times of a file.
- `set_xattr(hdfs_path, xattr_name, xattr_value, flags=0)`: Sets an extended attribute.
- `walk(top, topdown=True, onerror=None, followlinks=False)`: Generates file names in a directory tree by walking the tree top-down or bottom-up.
```

--------------------------------

### Create File in HDFS

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Use this method to create a new file at a specified path in HDFS. It supports uploading data from bytes or a file-like object and allows configuration of overwrite, block size, replication, permission, and buffer size.

```python
    def create(
        self,
        path: str,
        data: IO[bytes] | bytes,
        **kwargs: _PossibleArgumentTypes,
    ) -> None:
        """Create a file at the given path.

        :param data: ``bytes`` or a ``file``-like object to upload
        :param overwrite: If a file already exists, should it be overwritten?
        :type overwrite: bool
        :param blocksize: The block size of a file.
        :type blocksize: long
        :param replication: The number of replications of a file.
        :type replication: short
        :param permission: The permission of a file/directory. Any radix-8 integer (leading zeros
            may be omitted.)
        :type permission: octal
        :param buffersize: The size of the buffer used in transferring data.
        :type buffersize: int
        """
        metadata_response = self._put(
            path, "CREATE", expected_status=HTTPStatus.TEMPORARY_REDIRECT, **kwargs
        )
        assert not metadata_response.content
        data_response = self._requests_session.put(
            metadata_response.headers["location"], data=data, **self._requests_kwargs
        )
        _check_response(data_response, expected_status=HTTPStatus.CREATED)
        assert not data_response.content

```

--------------------------------

### pyhdfs.HdfsClient Methods

Source: https://pyhdfs.readthedocs.io/en/latest/genindex.html

Methods available on the HdfsClient class for interacting with HDFS.

```APIDOC
## append()
### Description
Appends data to an existing file.
### Method
append
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## concat()
### Description
Concatenates multiple files into a single file.
### Method
concat
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## copy_from_local()
### Description
Copies a file from the local filesystem to HDFS.
### Method
copy_from_local
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## copy_to_local()
### Description
Copies a file from HDFS to the local filesystem.
### Method
copy_to_local
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## create()
### Description
Creates a new file in HDFS.
### Method
create
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## create_snapshot()
### Description
Creates a snapshot of a directory.
### Method
create_snapshot
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## create_symlink()
### Description
Creates a symbolic link.
### Method
create_symlink
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## delete()
### Description
Deletes a file or directory from HDFS.
### Method
delete
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## delete_snapshot()
### Description
Deletes a snapshot of a directory.
### Method
delete_snapshot
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## exists()
### Description
Checks if a file or directory exists in HDFS.
### Method
exists
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_active_namenode()
### Description
Retrieves the active NameNode.
### Method
get_active_namenode
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_content_summary()
### Description
Gets the ContentSummary for a given path.
### Method
get_content_summary
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_file_checksum()
### Description
Gets the checksum of a file.
### Method
get_file_checksum
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_file_status()
### Description
Gets the FileStatus for a given path.
### Method
get_file_status
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_home_directory()
### Description
Retrieves the user's home directory.
### Method
get_home_directory
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## get_xattrs()
### Description
Gets the extended attributes for a given path.
### Method
get_xattrs
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## list_status()
### Description
Lists the status of files and directories in a given path.
### Method
list_status
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## list_xattrs()
### Description
Lists the extended attributes for a given path.
### Method
list_xattrs
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## listdir()
### Description
Lists the contents of a directory.
### Method
listdir
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## mkdirs()
### Description
Creates directories in HDFS.
### Method
mkdirs
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## open()
### Description
Opens a file in HDFS for reading or writing.
### Method
open
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## remove_xattr()
### Description
Removes an extended attribute from a given path.
### Method
remove_xattr
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## rename()
### Description
Renames a file or directory in HDFS.
### Method
rename
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## rename_snapshot()
### Description
Renames a snapshot of a directory.
### Method
rename_snapshot
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## set_owner()
### Description
Sets the owner of a file or directory.
### Method
set_owner
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## set_permission()
### Description
Sets the permission of a file or directory.
### Method
set_permission
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## set_replication()
### Description
Sets the replication factor of a file.
### Method
set_replication
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## set_times()
### Description
Sets the access and modification times of a file.
### Method
set_times
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## set_xattr()
### Description
Sets an extended attribute for a given path.
### Method
set_xattr
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

```APIDOC
## walk()
### Description
Generates file names in a directory tree by walking the tree top-down or bottom-up.
### Method
walk
### Endpoint
N/A (Method call)
### Parameters
N/A
### Request Example
N/A
### Response
N/A
```

--------------------------------

### Copy File to Local Filesystem

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Copies a single file from HDFS to the local filesystem. It utilizes the `open` method for reading from HDFS and standard file operations for writing locally. Accepts arguments compatible with the `open` method.

```python
def copy_to_local(
        self,
        src: str,
        localdest: str,
        **kwargs: _PossibleArgumentTypes
    ) -> None:
        """Copy a single file from ``src`` to the local file system

        Takes all arguments that :py:meth:`open` takes.
        """
        with self.open(src, **kwargs) as fsrc:
            with open(localdest, "wb") as fdst:
                shutil.copyfileobj(fsrc, fdst)
```

--------------------------------

### Walk HDFS Directory Tree

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Recursively walks a directory tree in HDFS, similar to os.walk. Yields directory names, sub-directory names, and file names.

```python
try:
            listing = self.list_status(top, **kwargs)
        except HdfsException as e:
            if onerror is not None:
                onerror(e)
            return

        dirnames, filenames = [], []
        for f in listing:
            if f.type == "DIRECTORY":
                dirnames.append(f.pathSuffix)
            elif f.type == "FILE":
                filenames.append(f.pathSuffix)
            else:  # pragma: no cover
                raise AssertionError(f"Unexpected type {f.type}")

        if topdown:
            yield top, dirnames, filenames
        for name in dirnames:
            new_path = posixpath.join(top, name)
            yield from self.walk(new_path, topdown, onerror, **kwargs)
        if not topdown:
            yield top, dirnames, filenames
```

--------------------------------

### copy_from_local

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Copies a single file from the local file system to a specified destination in HDFS. Accepts arguments similar to the `create()` method.

```APIDOC
## copy_from_local

### Description
Copy a single file from the local file system to `dest`.
Takes all arguments that `create()` takes.

### Parameters
* **_localsrc** (str) - The path to the local source file.
* **_dest** (str) - The destination path in HDFS.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.
```

--------------------------------

### listdir

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Return a list containing names of files in the given path.

```APIDOC
## listdir

### Description
Return a list containing names of files in the given path.

### Method
listdir

### Parameters
#### Path Parameters
- **_path** (str) - The path to list files from.

#### Other Parameters
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
list
```

--------------------------------

### get_content_summary

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Return the ContentSummary of a given Path.

```APIDOC
## get_content_summary

### Description
Return the :py:class:`ContentSummary` of a given Path.

### Method
GET

### Endpoint
/{path}?op=GETCONTENTSUMMARY

### Parameters
#### Path Parameters
- **path** (str) - Required - The HDFS path to get the content summary for.

### Response
#### Success Response (200)
- **ContentSummary** - An object containing the content summary information.
```

--------------------------------

### Copy File from Local to HDFS

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Copies a single file from the local file system to a specified destination in HDFS.

```python
def copy_from_local(
        self, localsrc: str, dest: str, **kwargs: _PossibleArgumentTypes
    ) -> None:
        """Copy a single file from the local file system to ``dest``

```

--------------------------------

### copy_to_local

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Copies a single file from HDFS to a specified destination on the local file system. Accepts arguments similar to the `open()` method.

```APIDOC
## copy_to_local

### Description
Copy a single file from `src` to the local file system.
Takes all arguments that `open()` takes.

### Parameters
* **_src** (str) - The source path in HDFS.
* **_localdest** (str) - The destination path on the local file system.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.
```

--------------------------------

### set_permission

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Set permission of a path.

```APIDOC
## set_permission

### Description
Set permission of a path.

### Method
set_permission

### Parameters
#### Path Parameters
- **_path** (str) - The path of the file or directory.

#### Other Parameters
- **permission** (octal) - The permission of a file/directory. Any radix-8 integer (leading zeros may be omitted.).
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
None
```

--------------------------------

### Make WebHDFS Request

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Handles WebHDFS requests, including NameNode failover and error checking. All keyword arguments are passed as query parameters. Ensures the path is absolute and sets the default user.name.

```python
def _request(
        self,
        method: str,
        path: str,
        op: str,
        expected_status: HTTPStatus,
        **kwargs: _PossibleArgumentTypes,
    ) -> requests.Response:
        """Make a WebHDFS request against the NameNodes

        This function handles NameNode failover and error checking.
        All kwargs are passed as query params to the WebHDFS server.
        """
        hosts = self.hosts
        if not posixpath.isabs(path):
            raise ValueError(f"Path must be absolute, was given {path}")
        _transform_user_name_key(kwargs)
        kwargs.setdefault("user.name", self.user_name)

        formatted_args = " ".join("{}={}".format(*t) for t in kwargs.items())
        _logger.info("%s %s %s %s", op, path, formatted_args, ",".join(hosts))
        kwargs["op"] = op
        for i in range(self.max_tries):
            log_level = logging.DEBUG if i < self.max_tries - 1 else logging.WARNING
            for host in hosts:
                try:
                    response = self._requests_session.request(
                        method,
                        f"{self.scheme}://{host}{WEBHDFS_PATH}{url_quote(path.encode('utf-8'))}",
                        params=kwargs,
                        timeout=self.timeout,
```

--------------------------------

### exists

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Return true if the given path exists.

```APIDOC
## exists

### Description
Return true if the given path exists.

### Method
GET (assumed based on internal get_file_status call)

### Endpoint
/{path}?op=GETFILESTATUS

### Parameters
#### Path Parameters
- **path** (str) - The path to check for existence.

#### Query Parameters
- **kwargs** (_PossibleArgumentTypes) - Additional keyword arguments.

### Response
#### Success Response (200)
- **bool** - True if the path exists, False otherwise.

### Throws
- **HdfsFileNotFoundException** - If the path does not exist.
```

--------------------------------

### set_xattr

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Set an xattr of a file or directory.

```APIDOC
## set_xattr

### Description
Set an xattr of a file or directory.

### Method
set_xattr

### Parameters
#### Path Parameters
- **_path** (str) - The path of the file or directory.
- **xattr_name** (str) - The name must be prefixed with the namespace followed by `.`. For example, `user.attr`.
- **xattr_value** (str | None) - The value of the xattr.
- **flag** (str) - `CREATE` or `REPLACE`.

#### Other Parameters
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
None
```

--------------------------------

### rename

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Renames Path src to Path dst.

```APIDOC
## rename

### Description
Renames Path src to Path dst.

### Method
rename

### Parameters
#### Path Parameters
- **_path** (str) - The source path.
- **destination** (str) - The destination path.

#### Other Parameters
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Returns
true if rename is successful

### Return Type
bool
```

--------------------------------

### List Directory Contents

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Returns a list of file and directory names within a given path in HDFS. Raises NotADirectoryError if the path is a file.

```python
statuses = self.list_status(path, **kwargs)
        if (
            len(statuses) == 1
            and statuses[0].pathSuffix == ""
            and statuses[0].type == "FILE"
        ):
            raise NotADirectoryError(f"Not a directory: {path!r}")
        return [f.pathSuffix for f in statuses]
```

--------------------------------

### set_replication

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Set replication for an existing file.

```APIDOC
## set_replication

### Description
Set replication for an existing file.

### Method
set_replication

### Parameters
#### Path Parameters
- **_path** (str) - The path of the file.

#### Other Parameters
- **replication** (short) - new replication.
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Returns
true if successful; false if file does not exist or is a directory

### Return Type
bool
```

--------------------------------

### get_content_summary

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Retrieves a `ContentSummary` object for a given path in HDFS, providing information like file count, directory count, and total size.

```APIDOC
## get_content_summary

### Description
Return the `ContentSummary` of a given Path.

### Parameters
* **_path** (str) - The path for which to get the content summary.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.

### Returns
* **ContentSummary** - An object containing the content summary of the path.
```

--------------------------------

### append

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Appends data to an existing file at the specified path. Supports specifying a buffer size for the operation.

```APIDOC
## append

### Description
Append to the given file.

### Parameters
#### Path Parameters
- **path** (str) - The path to the file to append to.
- **data** (bytes | IO[bytes]) - The data to append, can be bytes or a file-like object.

#### Keyword Arguments
- **buffersize** (int) - The size of the buffer used in transferring data.
```

--------------------------------

### pyhdfs.ContentSummary

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Represents the content summary of a directory in HDFS. It provides details on the number of directories, files, bytes consumed, and various quota information.

```APIDOC
class pyhdfs.ContentSummary(_** kwargs: object_)

    Bases: `_BoilerplateClass`

    Parameters:
        
      * **directoryCount** (_int_) – The number of directories.
      * **fileCount** (_int_) – The number of files.
      * **length** (_int_) – The number of bytes used by the content.
      * **quota** (_int_) – The namespace quota of this directory.
      * **spaceConsumed** (_int_) – The disk space consumed by the content.
      * **spaceQuota** (_int_) – The disk space quota.
      * **typeQuota** (_Dict_ _[__str_ _,__TypeQuota_ _]_) – Quota usage for ARCHIVE, DISK, SSD

directoryCount: int
     
fileCount: int
     
length: int
     
quota: int
     
spaceConsumed: int
     
spaceQuota: int
     
typeQuota: dict[str, TypeQuota]
     
```

--------------------------------

### ContentSummary Data Structure

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Provides a summary of HDFS content, including directory and file counts, space consumed, and various quota details.

```APIDOC
## ContentSummary

### Description
Provides a summary of HDFS content.

### Parameters
- **directoryCount** (int) - The number of directories.
- **fileCount** (int) - The number of files.
- **length** (int) - The number of bytes used by the content.
- **quota** (int) - The namespace quota of this directory.
- **spaceConsumed** (int) - The disk space consumed by the content.
- **spaceQuota** (int) - The disk space quota.
- **typeQuota** (Dict[str, TypeQuota]) - Quota usage for ARCHIVE, DISK, SSD.
```

--------------------------------

### append

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Appends data to an existing file in HDFS. Supports both bytes and file-like objects for data input.

```APIDOC
## append

### Description
Appends to the given file.

### Parameters
* **_path** (str) - The path to the file to append to.
* **_data** (bytes | IO[bytes]) - The data to append, can be bytes or a file-like object.
* **buffersize** (int) - The size of the buffer used in transferring data.
* **_**kwargs: str | int | None | list[str] - Additional keyword arguments.
```

--------------------------------

### set_times

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Set access time of a file.

```APIDOC
## set_times

### Description
Set access time of a file.

### Method
set_times

### Parameters
#### Path Parameters
- **_path** (str) - The path of the file.

#### Other Parameters
- **modificationtime** (long) - Set the modification time of this file. The number of milliseconds since Jan 1, 1970.
- **accesstime** (long) - Set the access time of this file. The number of milliseconds since Jan 1 1970.
- **_kwargs** (str | int | None | list[str]) - Additional keyword arguments.

### Return Type
None
```

--------------------------------

### ContentSummary Class

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

Represents the summary of content for a file or directory in HDFS.

```APIDOC
## ContentSummary Class

### Description
Provides attributes that summarize the content of a file or directory, such as counts of files, directories, and total size.

### Attributes
- `directoryCount` (int): The number of directories.
- `fileCount` (int): The number of files.
- `length` (int): The total size of the content in bytes.
- `quota` (int): The quota set for the directory.
- `spaceConsumed` (int): The space consumed by the content in bytes.
- `spaceQuota` (int): The space quota set for the directory.
- `typeQuota` (TypeQuota): The type quota information.
```

--------------------------------

### pyhdfs.HdfsRuntimeException

Source: https://pyhdfs.readthedocs.io/en/latest/pyhdfs.html

General runtime exception for HDFS operations. Inherits from HdfsHttpException.

```APIDOC
## Exception: pyhdfs.HdfsRuntimeException

### Description
General runtime exception for HDFS operations. Inherits from HdfsHttpException.

### Parameters
* **message** – Exception message
* **exception** – Name of the exception
* **status_code** (_int_) – HTTP status code
* **kwargs** – any extra attributes in case Hadoop adds more stuff
```

--------------------------------

### Parse HDFS Hosts

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Parses a comma or semicolon-separated string of hosts or an iterable of hosts into a list. Appends the default port if not specified. Shuffles the list if randomization is enabled.

```python
def _parse_hosts(self, hosts: str | Iterable[str]) -> list[str]:
        host_list = re.split(r",|;", hosts) if isinstance(hosts, str) else list(hosts)
        for i, host in enumerate(host_list):
            if ":" not in host:
                host_list[i] = f"{host:s}:{DEFAULT_PORT:d}"
        if self.randomize_hosts:
            random.shuffle(host_list)
        return host_list
```

--------------------------------

### set_replication

Source: https://pyhdfs.readthedocs.io/en/latest/_modules/pyhdfs.html

Sets the replication factor for an existing file in HDFS. Returns true if successful, false otherwise.

```APIDOC
## set_replication

### Description
Set replication for an existing file.

### Method Signature
`set_replication(self, path: str, **kwargs: _PossibleArgumentTypes) -> bool`

### Parameters
* `path` (str): The path to the file.
* `replication` (short): The new replication factor.
* `**kwargs`: Additional keyword arguments.

### Returns
* `bool`: True if successful; false if file does not exist or is a directory.
```
Tokens:
Raw