HdfsCLI

https://hdfscli.readthedocs.io/en/latest/
Admin
HdfsCLI provides an API and command-line interface for interacting with HDFS, allowing users to...
Tokens:31,947
Snippets:266
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### Exploring the File System

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Provides examples of common file system operations such as getting content summary, listing directory contents, retrieving file status, renaming files, and deleting files or folders.

```APIDOC
## Exploring the File System

### Retrieving a file or folder content summary

```python
content = client.content('dat')
```

### Listing all files inside a directory

```python
fnames = client.list('dat')
```

### Retrieving a file or folder status

```python
status = client.status('dat/features')
```

### Renaming ("moving") a file

```python
client.rename('dat/features', 'features')
```

### Deleting a file or folder

```python
client.delete('dat', recursive=True)
```

### Downloading a file or folder locally

```python
client.download('dat', 'dat', n_threads=5)
```

### Getting all files under a given folder (arbitrary depth)

```python
import posixpath as psp
fpaths = [
  psp.join(dpath, fname)
  for dpath, _, fnames in client.walk('predictions')
  for fname in fnames
]
```
```

--------------------------------

### Install HdfsCLI

Source: https://hdfscli.readthedocs.io/en/latest/index.html

Install the HdfsCLI package using pip.

```bash
$ pip install hdfs

```

--------------------------------

### Install HdfsCLI with Extensions

Source: https://hdfscli.readthedocs.io/en/latest/index.html

Install HdfsCLI with specific extensions like avro, dataframe, or kerberos using pip.

```bash
$ pip install hdfs[avro,dataframe,kerberos]

```

--------------------------------

### Configure HDFS CLI Logging

Source: https://hdfscli.readthedocs.io/en/latest/advanced.html

Example configuration for setting the log level and log file path for a specific HDFS CLI entry point, such as 'hdfscli-avro'.

```ini
[hdfscli-avro.command]
log.level = INFO
log.path = /tmp/hdfscli/avro.log

```

--------------------------------

### Configure and Get Log Handler

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Configures and returns a log handler for a given command. It looks up options in the '[COMMAND.command]' section. Returns a TimedRotatingFileHandler or a NullHandler if logging is disabled.

```python
from hdfs import Config

config = Config()
log_handler = config.get_log_handler('my_command')
```

--------------------------------

### walk

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Performs a depth-first traversal of the remote filesystem starting from a given path. Can optionally return file status information and control traversal depth.

```APIDOC
## walk

### Description
Performs a depth-first traversal of the remote filesystem starting from a given path. Can optionally return file status information and control traversal depth.

### Method
Not specified (likely internal SDK method)

### Endpoint
Not specified

### Parameters
#### Path Parameters
None

#### Query Parameters
- **depth** (integer) - Optional - Maximum depth to explore. `0` for no limit.
- **status** (boolean) - Optional - If true, also return each file or folder's corresponding FileStatus_.
- **ignore_missing** (boolean) - Optional - Ignore missing nested folders rather than raise an exception.
- **allow_dir_changes** (boolean) - Optional - Allow changes to the directories' list to affect the walk.

#### Request Body
None

### Request Example
None

### Response
#### Success Response
- **generator** - Yields tuples `(path, dirs, files)` where `path` is the absolute path to the current directory, `dirs` is the list of directory names it contains, and `files` is the list of file names it contains.

#### Response Example
```python
# Example of yielded tuple
('/path/to/dir', ['subdir1', 'subdir2'], ['file1.txt', 'file2.txt'])
```
```

--------------------------------

### AvroWriter Start Writer

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/avro.html

Internal method to start the underlying Avro writer, typically called when the schema is available.

```python
def _start_writer(self):
    _logger.debug('Starting underlying writer.')
```

--------------------------------

### Interactive HDFS Shell with HdfsCLI

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Start an interactive HDFS client session using `hdfscli --alias=dev`. The client is available as `CLIENT` within the IPython shell for performing file system operations.

```bash
$ hdfscli --alias=dev

Welcome to the interactive HDFS python shell.
The HDFS client is available as `CLIENT`.

In [1]: CLIENT.list('data/')
Out[1]: ['1.json', '2.json']

In [2]: CLIENT.status('data/2.json')
Out[2]: {
  'accessTime': 1439743128690,
  'blockSize': 134217728,
  'childrenNum': 0,
  'fileId': 16389,
  'group': 'supergroup',
  'length': 2,
  'modificationTime': 1439743129392,
  'owner': 'drwho',
  'pathSuffix': '',
  'permission': '755',
  'replication': 1,
  'storagePolicy': 0,
  'type': 'FILE'
}

In [3]: CLIENT.delete('data/2.json')
Out[3]: True
```

--------------------------------

### Get HDFS Path Parts

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Returns a dictionary of part-files corresponding to a given HDFS path. This is useful for understanding the internal structure of HDFS files, especially for partitioned data.

```python
client.parts('foo')

```

--------------------------------

### Write to an HDFS File

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Use the `write()` method to get a file-like writable object for writing data to HDFS. Ensure the connection is closed using a `with` block.

```python
# Writing part of a file.
with open('samples') as reader, client.write('samples') as writer:
  for line in reader:
    if line.startswith('-'):
      writer.write(line)
```

--------------------------------

### Set HDFSCLI_ENTRY_POINT Environment Variable

Source: https://hdfscli.readthedocs.io/en/latest/advanced.html

Specify the HDFSCLI_ENTRY_POINT environment variable before installing hdfscli to change the command-line entry point name. This affects both the main entry point and extension prefixes.

```bash
$ HDFSCLI_ENTRY_POINT=hdfs pip install hdfs
```

--------------------------------

### Explore HDFS File System

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Interact with HDFS using various client methods to get content summaries, list directories, retrieve status, rename files, and delete items.

```python
# Retrieving a file or folder content summary.
content = client.content('dat')

# Listing all files inside a directory.
fnames = client.list('dat')

# Retrieving a file or folder status.
status = client.status('dat/features')

# Renaming ("moving") a file.
client.rename('dat/features', 'features')

# Deleting a file or folder.
client.delete('dat', recursive=True)
```

--------------------------------

### Implement a Secure Client Subclass

Source: https://hdfscli.readthedocs.io/en/latest/advanced.html

Define a custom client subclass that extends the base Client class to handle specific connection requirements, such as HTTPS with custom certificate and verification options. This example demonstrates passing a custom session object.

```python
from hdfs import Client
from requests import Session

class SecureClient(Client):

  """A new client subclass for handling HTTPS connections.

  :param url: URL to namenode.
  :param cert: Local certificate. See `requests` documentation for details
    on how to use this.
  :param verify: Whether to check the host's certificate.
  :param **kwargs: Keyword arguments passed to the default `Client`
    constructor.

  """

  def __init__(self, url, cert=None, verify=True, **kwargs):
    session = Session()
    if ',' in cert:
      session.cert = [path.strip() for path in cert.split(',')]
    else:
      session.cert = cert
    if isinstance(verify, basestring): # Python 2.
      verify = verify.lower() in ('true', 'yes', 'ok')
    session.verify = verify
    super(SecureClient, self).__init__(url, session=session, **kwargs)
```

--------------------------------

### Get HDFS Client by Alias

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/config.html

Retrieves an HDFS client instance based on an alias. If no alias is provided, it attempts to use the default alias specified in the global configuration. Subsequent calls with the same alias return the cached client instance.

```python
def get_client(self, alias=None):
    """Load HDFS client.

    :param alias: The client to look up. If not specified, the default alias be
      used (`default.alias` option in the `global` section) if available and an
      error will be raised otherwise.

    Further calls to this method for the same alias will return the same client
    instance (in particular, any option changes to this alias will not be taken
    into account).

    """
    if not alias:
      if (
        not self.has_section(self.global_section) or
        not self.has_option(self.global_section, 'default.alias')
      ):
        raise HdfsError('No alias specified and no default alias found.')
      alias = self.get(self.global_section, 'default.alias')
    if not alias in self._clients:
      for suffix in ('.alias', '_alias'):
        section = '{}{}'.format(alias, suffix)
        if self.has_section(section):
          options = dict(self.items(section))
          class_name = options.pop('client', 'InsecureClient')
          # Massage options.
          if 'timeout' in options:
            timeout = tuple(int(s) for s in options['timeout'].split(','))
            options['timeout'] = timeout[0] if len(timeout) == 1 else timeout
          self._clients[alias] = Client.from_options(options, class_name)
          break
      else:
        raise HdfsError('Alias %r not found in %r.', alias, self.path)
    return self._clients[alias]
```

--------------------------------

### Instantiating a Client

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Demonstrates two ways to instantiate an HDFS client: directly using the constructor or by loading configuration from a file.

```APIDOC
## Instantiating a Client

### Option 1: Direct Constructor Call

This is the most straightforward and flexible method.

```python
from hdfs import InsecureClient
client = InsecureClient('http://host:port', user='ann')
```

### Option 2: Using Configuration File

This method leverages the `hdfs.config.Config` class to load an existing configuration file and create clients from existing aliases.

```python
from hdfs import Config
client = Config().get_client('dev')
```
```

--------------------------------

### KerberosClient Initialization

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Demonstrates how to initialize the KerberosClient for connecting to a Kerberized HDFS cluster.

```APIDOC
## KerberosClient Initialization

### Description
Initializes the `KerberosClient` for connecting to HDFS clusters secured with Kerberos authentication.

### Method
Constructor

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from hdfs.ext.kerberos import KerberosClient

client = KerberosClient('http://host:port')
```

### Response
None (initialization)

#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Get Current Microseconds Helper

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Helper function to get the current time in microseconds as a string.

```python
def _current_micros():
  """Returns a string representing the current time in microseconds."""
  return str(int(time.time() * 1e6))
```

--------------------------------

### Initialize KerberosClient

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Instantiate the KerberosClient for connecting to a Kerberized HDFS cluster. Ensure the HDFS URL is correctly specified.

```python
from hdfs.ext.kerberos import KerberosClient
client = KerberosClient('http://host:port')
```

--------------------------------

### Instantiate HDFS Client with Config

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Leverage existing configuration files to create an HDFS client instance. This is useful for reusing configured aliases.

```python
from hdfs import Config
client = Config().get_client('dev')
```

--------------------------------

### Instantiate Client from Options

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Load an HDFS client instance using a dictionary of options. This method allows for flexible client configuration and can instantiate any registered Client subclass.

```python
client = Client.from_options(options)

```

--------------------------------

### Client Initialization

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Initializes the HDFS client with connection details and optional configurations.

```APIDOC
## Client(url, root=None, proxy=None, timeout=None, session=None)

### Description
Initializes the HDFS client. This client should generally be used directly only when its subclasses do not offer sufficient flexibility.

### Parameters
- **url** (string) - Required - Hostname or IP address of HDFS namenode, prefixed with protocol, followed by WebHDFS port on namenode. Can specify multiple URLs separated by semicolons for High Availability.
- **root** (string) - Optional - Root path, prefixed to all HDFS paths. If relative, it's assumed relative to the user's home directory.
- **proxy** (string) - Optional - User to proxy as.
- **timeout** (float or tuple) - Optional - Connection timeouts, forwarded to the request handler. Can be a float for a single timeout or a tuple of (connect_timeout, read_timeout).
- **session** (`requests.Session`) - Optional - A `requests.Session` instance to use for all requests.
```

--------------------------------

### from_options

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Class method to instantiate a Client instance from a dictionary of options. Allows for flexible client configuration.

```APIDOC
## from_options

### Description
Load client from options.

### Parameters
* **options** (dict) - Options dictionary.
* **class_name** (string, optional) - Client class name. Defaults to 'Client'.

### Returns
* Client - An instance of the Client class.
```

--------------------------------

### Show hdfscli-avro help

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/avro.html

Use the hdfscli-avro command-line entry point to display help information.

```bash
$ hdfscli-avro --help
```

--------------------------------

### Stream File Content in Chunks

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

When dealing with large files, use the `read()` method with a `chunk_size` argument to get a generator for streaming file contents efficiently.

```python
# Stream a file.
with client.read('features', chunk_size=8096) as reader:
  for chunk in reader:
    pass
```

--------------------------------

### Instantiate KerberosClient

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/kerberos.html

Use KerberosClient to connect to a Kerberized HDFS cluster. Ensure the module is loaded in your hdfscli.cfg.

```python
from hdfs.ext.kerberos import KerberosClient
client = KerberosClient('http://host:port')
```

--------------------------------

### HDFS Client Initialization

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Initializes the HDFS client with connection details and optional configurations. It sets up the session, proxy, and timeout for subsequent requests.

```python
def __init__(self, url, root=None, proxy=None, timeout=None, session=None):
    self.root = root
    self.url = url
    self.urls = [u for u in url.split(';') if u]
    self._urls = deque(self.urls) # this is rotated and used internally
    self._session = session or rq.Session()
    self._proxy = proxy
    self._timeout = timeout
    self._lock = Lock()
    _logger.info('Instantiated %r.', self)
```

--------------------------------

### Walk HDFS Directory Tree

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Recursively traverse a directory in HDFS to get all files within it using the `walk()` method. This is useful for processing all files under a given path.

```python
# Get all files under a given folder (arbitrary depth).
import posixpath as psp
fpaths = [
  psp.join(dpath, fname)
  for dpath, _, fnames in client.walk('predictions')
  for fname in fnames
]
```

--------------------------------

### Instantiate HDFS Client with InsecureClient

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Use this method to programmatically instantiate an HDFS client when security is not a concern. Ensure you have the correct host and port.

```python
from hdfs import InsecureClient
client = InsecureClient('http://host:port', user='ann')
```

--------------------------------

### KerberosClient Initialization

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/kerberos.html

Initializes the KerberosClient, handling Kerberos authentication with options for mutual authentication and concurrency limits. A session can be provided and will be modified in-place.

```python
def __init__(self, url, mutual_auth='OPTIONAL', max_concurrency=1, root=None,
    proxy=None, timeout=None, session=None, **kwargs):
    # We allow passing in a string as mutual authentication value.
    if isinstance(mutual_auth, string_types):
      try:
        mutual_auth = getattr(requests_kerberos, mutual_auth)
      except AttributeError:
        raise HdfsError('Invalid mutual authentication type: %r', mutual_auth)
    kwargs['mutual_authentication'] = mutual_auth
    if not session:
      session = rq.Session()
    session.auth = _HdfsHTTPKerberosAuth(int(max_concurrency), **kwargs)
    super(KerberosClient, self).__init__(
      url, root=root, proxy=proxy, timeout=timeout, session=session
    )
```

--------------------------------

### Client.from_options

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Loads an HDFS client instance from a dictionary of options. This class method provides a unified way to instantiate any registered Client subclass.

```APIDOC
## Client.from_options

### Description
Loads client from options. This class method provides a single entry point to instantiate any registered :class:`Client` subclass.

### Method
classmethod

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **options** (dict) - Required - Options dictionary.
* **class_name** (str) - Optional - Client class name. Defaults to the base :class:`Client` class.

### Request Example
```python
Client.from_options({'host': 'localhost', 'port': 8021}, class_name='Client')
```

### Response
#### Success Response (200)
An instance of a registered Client subclass.

#### Response Example
```json
{
  "client_instance": "<instance of Client subclass>"
}
```

### Error Handling
* **HdfsError**: Raised if the client class is unknown or if the options are invalid.
```

--------------------------------

### AvroWriter Context Manager

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/avro.html

Manages the AvroWriter within a `with` block. It ensures the underlying Avro writer is properly started (if schema is provided) and closed, and that the file handle is also closed. Reusing the writer within the same context is prevented.

```python
def __enter__(self):
    if self._entered:
      raise HdfsError('Avro writer cannot be reused.')
    self._entered = True
    if self._schema:
      self._start_writer()
    return self
```

```python
def __exit__(self, *exc_info):
    if not self._writer:
      return # No header or records were written.
    try:
      self._writer.__exit__(*exc_info)
      _logger.debug('Closed underlying writer.')
    finally:
      self._fo.__exit__(*exc_info)
```

--------------------------------

### Initialize Insecure HDFS Client

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Instantiate an `InsecureClient` for connecting to HDFS when security is disabled. Provide the HDFS namenode URL, including the protocol and port. The `user` parameter can be specified, otherwise, it defaults to the current system user.

```python
client = InsecureClient('http://namenode:50070', user='test')
```

--------------------------------

### Token HDFS Client Initialization

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Initializes an HDFS web client using Hadoop token delegation security, setting the delegation token in the session.

```python
class TokenClient(Client):

  r"""HDFS web client using Hadoop token delegation security.

  :param url: Hostname or IP address of HDFS namenode, prefixed with protocol,
    followed by WebHDFS port on namenode
  :param token: Hadoop delegation token.
  :param **kwargs: Keyword arguments passed to the base class' constructor.

  Note that if a session argument is passed in, it will be modified in-place to
  support authentication.

  """

  def __init__(self, url, token, **kwargs):
    session = kwargs.setdefault('session', rq.Session())
    if not session.params:
      session.params = {}
    session.params['delegation'] = token
    super(TokenClient, self).__init__(url, **kwargs)
```

--------------------------------

### Initialize AvroReader

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/avro.html

Initializes an AvroReader for a given HDFS path. It handles both single Avro files and directories containing multiple part-files. The reader_schema can be provided to specify the expected schema.

```python
def __init__(self, client, hdfs_path, parts=None, reader_schema=None):
    self.content = client.content(hdfs_path) #: Content summary of Avro file.
    self.metadata = None #: Avro header metadata.
    self.reader_schema = reader_schema #: Input reader schema.
    self._writer_schema = None
    if self.content['directoryCount']:
      # This is a folder.
      self._paths = [
        psp.join(hdfs_path, fname)
        for fname in client.parts(hdfs_path, parts)
      ]
    else:
      # This is a single file.
      self._paths = [hdfs_path]
    self._client = client
    self._records = None
    _logger.debug('Instantiated %r.', self)
```

--------------------------------

### Configure hdfscli.cfg for Kerberos

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/kerberos.html

Add 'hdfs.ext.kerberos' to 'autoload.modules' in your hdfscli.cfg to enable the KerberosClient.

```cfg
autoload.modules = hdfs.ext.kerberos
```

--------------------------------

### Instantiate HDFS Client from Alias

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Loads an HDFS client instance based on a specified alias. Subsequent calls for the same alias return the same instance. If no alias is provided, it defaults to 'default.alias' from the global section.

```python
from hdfs import Config

config = Config()
client = config.get_client('my_alias')
```

--------------------------------

### InsecureClient

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Initializes an HDFS web client for environments where security is disabled.

```APIDOC
## InsecureClient

### Description
HDFS web client to use when security is off.

### Parameters
* **url** (string) - Hostname or IP address of HDFS namenode, prefixed with protocol, followed by WebHDFS port.
* **user** (string, optional) - User default. Defaults to the current user.
* **kwargs** - Keyword arguments passed to the base class constructor.
```

--------------------------------

### Insecure HDFS Client Initialization

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Initializes an HDFS web client for use when security is off, setting the user and session parameters.

```python
class InsecureClient(Client):

  r"""HDFS web client to use when security is off.

  :param url: Hostname or IP address of HDFS namenode, prefixed with protocol,
    followed by WebHDFS port on namenode
  :param user: User default. Defaults to the current user's (as determined by
    `whoami`).
  :param **kwargs: Keyword arguments passed to the base class' constructor.

  Note that if a session argument is passed in, it will be modified in-place to
  support authentication.

  """

  def __init__(self, url, user=None, **kwargs):
    user = user or getuser()
    session = kwargs.setdefault('session', rq.Session())
    if not session.params:
      session.params = {}
    session.params['user.name'] = user
    super(InsecureClient, self).__init__(url, **kwargs)
```

--------------------------------

### Configure HDFS CLI with Kerberos Aliases

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Set up aliases in .hdfscli.cfg to specify the KerberosClient for different HDFS environments. This allows the CLI to automatically use the KerberosClient for the specified alias.

```ini
[global]
default.alias = dev
autoload.modules = hdfs.ext.kerberos

[dev.alias]
url = http://dev.namenode:port

[prod.alias]
url = http://prod.namenode:port
client = KerberosClient
```

--------------------------------

### HDFS Client File Reading Options

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Provides detailed options for reading files from HDFS, including chunk size, progress callbacks, and delimiter-based splitting. Raises errors for invalid configurations.

```python
if chunk_size < 0:
  raise ValueError('Read chunk size must be non-negative.')
if progress and not chunk_size:
  raise ValueError('Progress callback requires a positive chunk size.')
if delimiter:
  if not encoding:
    raise ValueError('Delimiter splitting requires an encoding.')
  if chunk_size:
    raise ValueError('Delimiter splitting incompatible with chunk size.')
_logger.info('Reading file %r.', hdfs_path)
res = self._open(
  hdfs_path,
  offset=offset,
  length=length,
  buffersize=buffer_size,
)
try:
  if not chunk_size and not delimiter:
    yield codecs.getreader(encoding)(res.raw) if encoding else res.raw
  else:
    res.encoding = encoding
    if delimiter:
      data = res.iter_lines(delimiter=delimiter, decode_unicode=True)
    else:
      data = res.iter_content(chunk_size=chunk_size, decode_unicode=True)
    if progress:
      def reader(_hdfs_path, _progress):
        """Generator that tracks progress."""
        nbytes = 0
        for chunk in data:
          nbytes += len(chunk)
          _progress(_hdfs_path, nbytes)
          yield chunk
        _progress(_hdfs_path, -1)
      yield reader(hdfs_path, progress)
    else:
      yield data
finally:
  res.close()
  _logger.debug('Closed response for reading file %r.', hdfs_path)
```

--------------------------------

### Display HDFS CLI Avro Help

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Shows the help message for the hdfscli-avro command-line entry point. Use this to understand available options for Avro file operations from the shell.

```bash
$ hdfscli-avro --help

```

--------------------------------

### Client Initialization

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Initializes a WebHDFS client to interact with the HDFS namenode. Supports high availability by specifying multiple URLs and allows for proxying and custom timeouts.

```APIDOC
## Client Initialization

### Description
Initializes a WebHDFS client to interact with the HDFS namenode. Supports high availability by specifying multiple URLs and allows for proxying and custom timeouts.

### Parameters
* **url** (string) - Required - Hostname or IP address of HDFS namenode, prefixed with protocol, followed by WebHDFS port on namenode. You may also specify multiple URLs separated by semicolons for High Availability support.
* **root** (string) - Optional - Root path, this will be prefixed to all HDFS paths passed to the client. If the root is relative, the path will be assumed relative to the user’s home directory.
* **proxy** (string) - Optional - User to proxy as.
* **timeout** (float or tuple) - Optional - Connection timeouts, forwarded to the request handler. How long to wait for the server to send data before giving up, as a float, or a `(connect_timeout, read_timeout)` tuple. If the timeout is reached, an appropriate exception will be raised. See the requests documentation for details.
* **session** (requests.Session) - Optional - `requests.Session` instance, used to emit all requests.
```

--------------------------------

### Autoload Modules and Paths

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/config.html

Loads modules and paths based on configuration options. Handles potential exceptions during loading and exits if errors occur.

```python
def _autoload(self):
    """Load modules to find clients."""

    def _load(suffix, loader):
      """Generic module loader."""
      option = 'autoload.{}'.format(suffix)
      if self.has_option(self.global_section, option):
        entries = self.get(self.global_section, option)
        for entry in entries.split(','):
          module = entry.strip()
          try:
            loader(module)
          except Exception: # pylint: disable=broad-except
            _logger.exception(
              'Unable to load %r defined at %r.',
              module, self.path
            )
            sys.exit(1)


    _load('modules', __import__)
    _load('paths', lambda path: load_source(
      osp.splitext(osp.basename(path))[0],
      path
    ))
```

--------------------------------

### Download Files and Folders from HDFS

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Use the `download()` method to copy files or entire folders from HDFS to your local file system. Supports multi-threaded downloads for performance.

```python
# Download a file or folder locally.
client.download('dat', 'dat', n_threads=5)
```

--------------------------------

### get_client

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Loads and returns an HDFS client instance based on an alias.

```APIDOC
## Method: Config.get_client

### Description
Loads an HDFS client. Subsequent calls with the same alias return the same instance.

### Parameters
* **alias** (string, optional) - The client alias to look up. Defaults to the `default.alias` option in the `global` section.
```

--------------------------------

### makedirs

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Creates a directory on HDFS, including any necessary parent directories. Supports setting permissions on new directories.

```APIDOC
## makedirs

### Description
Create a remote directory, recursively if necessary.

### Parameters
* **hdfs_path** (string) - Remote path to create.
* **permission** (string, optional) - Octal permission to set on the newly created directory.

### Returns
* None - This function currently has no return value.
```

--------------------------------

### Writing Files

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Illustrates how to write data to HDFS using the `write()` method, covering writing parts of a file, serializing JSON objects, and writing directly from an iterable.

```APIDOC
## Writing Files

### Writing part of a file

```python
with open('samples') as reader, client.write('samples') as writer:
  for line in reader:
    if line.startswith('-'):
      writer.write(line)
```

### Writing a serialized JSON object

```python
with client.write('model.json', encoding='utf-8') as writer:
  from json import dump
  dump(model, writer)
```

### Writing directly from an iterable `data` argument

```python
from json import dumps
client.write('model.json', dumps(model))
```
```

--------------------------------

### Upload and Download Files with HdfsCLI

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Use `hdfscli upload` to write files to HDFS and `hdfscli download` to read them. The `--threads` option controls parallelism. Use `-` as a path to stream content from stdin or to stdout.

```bash
$ hdfscli upload --alias=dev weights.json models/
$ hdfscli download export/results/ "results-$(date +%F)"
```

```bash
$ hdfscli download logs/1987-03-23.txt - >>logs
```

--------------------------------

### Checking Path Existence

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Explains the recommended method for checking path existence using `content()` or `status()` with `strict=False`, which returns `None` for missing paths.

```APIDOC
## Checking Path Existence

Most methods will raise an `HdfsError` if called on a missing path. The recommended way to check for path existence is using `content()` or `status()` with a `strict=False` argument.

```python
# Example using content()
content_result = client.content('some/path', strict=False)
if content_result is None:
    print('Path does not exist')

# Example using status()
status_result = client.status('some/other/path', strict=False)
if status_result is None:
    print('Path does not exist')
```
```

--------------------------------

### Initialize AvroWriter

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/avro.html

Initializes an AvroWriter for a given HDFS path. It accepts an optional schema, codec, sync interval, and other parameters forwarded to the underlying HDFS client write method. The writer can infer the schema from the first record if not provided.

```python
def __init__(self, client, hdfs_path, schema=None, codec=None,
    sync_interval=None, sync_marker=None, metadata=None, **kwargs):
    self._hdfs_path = hdfs_path
    self._fo = client.write(hdfs_path, **kwargs)
    self._schema = schema
    self._writer_kwargs = {
      'codec': codec or 'null',
      'metadata': metadata,
      'sync_interval': sync_interval or 1000 * SYNC_SIZE,
      'sync_marker': sync_marker or os.urandom(SYNC_SIZE),
    }
    self._entered = False
    self._writer = None
    _logger.info('Instantiated %r.', self)
```

--------------------------------

### parts

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Returns a dictionary of part-files corresponding to a given HDFS path. Allows selection of specific parts or random sampling.

```APIDOC
## parts

### Description
Returns a dictionary of part-files corresponding to a path.

### Parameters
* **hdfs_path** (string) - Remote path.
* **parts** (list or int, optional) - List of part-files numbers or total number of part-files to select. Defaults to all part-files.
* **status** (boolean, optional) - If True, also return each file’s corresponding FileStatus. Defaults to False.

### Returns
* dict - A dictionary of part-files.
```

--------------------------------

### Reading Files

Source: https://hdfscli.readthedocs.io/en/latest/quickstart.html

Shows how to read files from HDFS using the `read()` method, including loading entire files into memory, deserializing JSON, and streaming file contents in chunks or delimited lines.

```APIDOC
## Reading Files

### Loading a file into memory

```python
with client.read('features') as reader:
  features = reader.read()
```

### Directly deserializing a JSON object

```python
with client.read('model.json', encoding='utf-8') as reader:
  from json import load
  model = load(reader)
```

### Stream a file using `chunk_size`

```python
with client.read('features', chunk_size=8096) as reader:
  for chunk in reader:
    pass
```

### Stream a file using `delimiter`

```python
with client.read('samples.csv', encoding='utf-8', delimiter='\n') as reader:
  for line in reader:
    pass
```
```

--------------------------------

### write

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Creates a file on HDFS, supporting streaming uploads and optional overwriting.

```APIDOC
## write

### Description
Create a file on HDFS.

### Parameters
* **hdfs_path** (string) - Path where to create file.
* **data** (string | generator | file object | None, optional) - Contents of file to write. If None, returns a file-like object.
* **overwrite** (bool, optional) - Overwrite any existing file or directory. Defaults to False.
* **permission** (octal, optional) - Octal permission for the new file.
* **blocksize** (int, optional) - Block size of the file.
* **replication** (int, optional) - Number of replications of the file.
* **buffersize** (int, optional) - Size of upload buffer.
* **append** (bool, optional) - Append to a file. Defaults to False.
* **encoding** (string, optional) - Encoding for data serialization.

### Sample usages
```python
from json import dump, dumps

records = [
  {'name': 'foo', 'weight': 1},
  {'name': 'bar', 'weight': 2},
]

# As a context manager:
with client.write('data/records.jsonl', encoding='utf-8') as writer:
  dump(records, writer)

# Or, passing in a generator directly:
client.write('data/records.jsonl', data=dumps(records), encoding='utf-8')
```
```

--------------------------------

### Client Methods

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

The Client class provides methods for interacting with HDFS. These methods are dynamically generated based on the WebHDFS API operations.

```APIDOC
## Client Methods

The `Client` class dynamically exposes methods corresponding to WebHDFS operations. These methods are generated from `_Request` objects defined within the class.

### Example Usage

```python
from hdfs import InsecureClient

# Assuming HDFS is running on localhost:9870
client = InsecureClient('http://localhost:9870', user='your_user')

# Example: List directory contents
contents = client.list('/user/your_user')
print(contents)

# Example: Create a directory
client.makedirs('/user/your_user/new_dir')

# Example: Upload a file
with open('local_file.txt', 'rb') as f_in:
    with client.write('/user/your_user/remote_file.txt', encoding='utf-8') as f_out:
        f_out.write(f_in.read())

# Example: Download a file
with client.read('/user/your_user/remote_file.txt', encoding='utf-8') as f_in:
    with open('downloaded_file.txt', 'w') as f_out:
        f_out.write(f_in.read())
```

### Underlying Mechanism

Each method is essentially a wrapper around an HTTP request to the WebHDFS REST API. The `_ClientType` metaclass processes `_Request` instances, converting them into callable API handlers. The `to_method` function within `_Request` constructs these handlers, which manage retries, host rotation, and error handling.
```

--------------------------------

### HDFS Client Representation

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Provides a string representation of the HDFS client instance, showing its class name and the configured URL.

```python
def __repr__(self):
    return '<{}(url={!r})>'.format(self.__class__.__name__, self.url)
```

--------------------------------

### HDFS Client Class Loader

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Provides a class method to load client instances from options, supporting registered subclasses.

```python
  @classmethod
  def from_options(cls, options, class_name='Client'):
    """Load client from options.

    :param options: Options dictionary.
    :param class_name: Client class name. Defaults to the base :class:`Client`
      class.

    This method provides a single entry point to instantiate any registered
    :class:`Client` subclass. To register a subclass, simply load its
    containing module. If using the CLI, you can use the `autoload.modules` and
    `autoload.paths` options.

    """
    try:
      return cls.__registry__[class_name](**options)
    except KeyError:
      raise HdfsError('Unknown client class: %r', class_name)
    except TypeError:
      raise HdfsError('Invalid options: %r', options)
```

--------------------------------

### Configure HDFS CLI for Kerberos Module

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Add the hdfs.ext.kerberos module to the autoload.modules list in the global section of your .hdfscli.cfg file to enable Kerberos support in the command-line interface.

```ini
autoload.modules = hdfs.ext.kerberos
```

--------------------------------

### walk

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Performs a depth-first walk of the remote HDFS filesystem, yielding directory contents.

```APIDOC
## walk

### Description
Depth-first walk of remote filesystem.

### Parameters
* **hdfs_path** (string) - Starting path.
* **depth** (int, optional) - Maximum depth to explore. 0 for no limit.
* **status** (bool, optional) - Whether to return FileStatus. Defaults to False.
* **ignore_missing** (bool, optional) - Ignore missing nested folders. Defaults to False.
* **allow_dir_changes** (bool, optional) - Allow directory list changes to affect the walk. Defaults to False.

### Returns
* **generator** - Yields tuples of (path, dirs, files).
```

--------------------------------

### Configure Autoload Modules for Custom Clients

Source: https://hdfscli.readthedocs.io/en/latest/advanced.html

Specify modules to be automatically loaded by HDFS CLI in the global section of the configuration file. This makes custom client classes discoverable.

```ini
[global]
autoload.modules = hdfs.ext.kerberos
```

--------------------------------

### Configure and Return Log Handler

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/config.html

Configures and returns a log handler for a given command. It supports file logging via TimedRotatingFileHandler or disables logging by returning a NullHandler if logging is disabled for the command.

```python
def get_log_handler(self, command):
    """Configure and return log handler.

    :param command: The command to load the configuration for. All options will
      be looked up in the `[COMMAND.command]` section. This is currently only
      used for configuring the file handler for logging. If logging is disabled
      for the command, a :class:`NullHandler` will be returned, else a
      :class:`TimedRotatingFileHandler`.

    
```

--------------------------------

### Write Data to HDFS using Context Manager

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Use the `write` method as a context manager to stream data to a file in HDFS. This is useful for large datasets that should not be loaded entirely into memory. Ensure the file path is correctly specified and the encoding is set.

```python
from json import dump, dumps

records = [
  {'name': 'foo', 'weight': 1},
  {'name': 'bar', 'weight': 2},
]

# As a context manager:
with client.write('data/records.jsonl', encoding='utf-8') as writer:
  dump(records, writer)
```

--------------------------------

### Create Snapshot

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Creates a snapshot for a remote folder in HDFS where snapshotting has been allowed.

```APIDOC
## create_snapshot

### Description
Creates snapshot for a remote folder where it was allowed.

### Method Signature
`create_snapshot(hdfs_path, snapshotname=None)`

### Parameters
* **hdfs_path** (string) - Required - Remote path to a directory.
* **snapshotname** (string) - Optional - The name to assign to the snapshot.
```

--------------------------------

### Create Remote Directory

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Create a directory in HDFS, including any necessary intermediate directories. Permissions can be set for newly created directories.

```python
client.makedirs('foo')

```

--------------------------------

### Read File from HDFS

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Provides a context manager for reading files from HDFS. Supports reading the entire file, a specific byte range, or streaming data in chunks or by delimiter. Requires a 'with' block for usage.

```python
@contextmanager
def read(self, hdfs_path, offset=0, length=None, buffer_size=None,
  encoding=None, chunk_size=0, delimiter=None, progress=None):
  """Read a file from HDFS.

  :param hdfs_path: HDFS path.
  :param offset: Starting byte position.
  :param length: Number of bytes to be processed. `None` will read the entire
    file.
  :param buffer_size: Size of the buffer in bytes used for transferring the
    data. Defaults the the value set in the HDFS configuration.
  :param encoding: Encoding used to decode the request. By default the raw
    data is returned. This is mostly helpful in python 3, for example to
    deserialize JSON data (as the decoder expects unicode).
  :param chunk_size: If set to a positive number, the context manager will
    return a generator yielding every `chunk_size` bytes instead of a
    file-like object (unless `delimiter` is also set, see below).
  :param delimiter: If set, the context manager will return a generator
    yielding each time the delimiter is encountered. This parameter requires
    the `encoding` to be specified.
  :param progress: Callback function to track progress, called every
    `chunk_size` bytes (not available if the chunk size isn't specified). It
    will be passed two arguments, the path to the file being uploaded and the
    number of bytes transferred so far. On completion, it will be called once
    with `-1` as second argument.

  This method must be called using a `with` block:

  .. code-block:: python

  
```

--------------------------------

### Config Class

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Manages HdfsCLI configuration settings, allowing programmatic access to configuration files and environment variables.

```APIDOC
## Class: hdfs.config.Config

### Description
Configuration class that inherits from `configparser.RawConfigParser`. It handles loading configuration from a specified path or environment variables.

### Parameters
* **path** (string, optional) - Path to the configuration file. Defaults to `~/.hdfscli.cfg` or the `HDFSCLI_CONFIG` environment variable.
* **stream_log_level** (any, optional) - Stream handler log level for the root logger. A false-ish value disables the handler.
```

--------------------------------

### create_snapshot

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Creates a snapshot for a remote folder on HDFS where snapshotting has been allowed. The server can generate a snapshot name if one is not provided. Returns the path to the created snapshot.

```APIDOC
## create_snapshot

### Description
Create snapshot for a remote folder where it was allowed.

### Parameters
* **hdfs_path** (string) - Required - Remote path to a direcotry. If `hdfs_path` doesn’t exist, doesn’t allow to create snapshot or points to a normal file, an `HdfsError` will be raised.
* **snapshotname** (string) - Optional - snapshot name; if absent, name is generated by the server.

### Returns
A path to created snapshot.
```

--------------------------------

### Make Directories

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/client.html

Creates a remote directory in HDFS, creating intermediate directories as necessary. An optional permission can be set for newly created directories.

```APIDOC
## makedirs

### Description
Creates a remote directory, recursively if necessary.

### Method Signature
`makedirs(hdfs_path, permission=None)`

### Parameters
* **hdfs_path** (string) - Required - Remote path. Intermediate directories will be created appropriately.
* **permission** (string) - Optional - Octal permission to set on the newly created directory. These permissions will only be set on directories that do not already exist.
```

--------------------------------

### Read Avro Files from HDFS

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Demonstrates how to read Avro files from HDFS using AvroReader. This allows for streaming decoding of records and accessing file metadata.

```python
with AvroReader(client, 'foo.avro') as reader:
  schema = reader.writer_schema # The remote file's Avro schema.
  content = reader.content # Content metadata (e.g. size).
  for record in reader:
    pass # and its records

```

--------------------------------

### Configure KerberosClient for a specific alias

Source: https://hdfscli.readthedocs.io/en/latest/_modules/hdfs/ext/kerberos.html

Specify 'client = KerberosClient' for an alias in hdfscli.cfg to use Kerberos authentication for that alias.

```cfg
[global]
default.alias = dev
autoload.modules = hdfs.ext.kerberos

[dev.alias]
url = http://dev.namenode:port

[prod.alias]
url = http://prod.namenode:port
client = KerberosClient
```

--------------------------------

### content

Source: https://hdfscli.readthedocs.io/en/latest/api.html

Fetches the ContentSummary for a file or folder on HDFS, providing details like file count, directory count, and total size. Can optionally return None if the path does not exist.

```APIDOC
## content

### Description
Get ContentSummary for a file or folder on HDFS.

### Parameters
* **hdfs_path** (string) - Required - Remote path.
* **strict** (bool) - Optional - If `False`, return `None` rather than raise an exception if the path doesn’t exist.
```

--------------------------------

### Implement Transfer Progress Tracker

Source: https://hdfscli.readthedocs.io/en/latest/advanced.html

A basic progress tracker callback class that logs the total transferred bytes for each file upon completion. It uses a lock to ensure thread-safe updates to the transfer data.

```python
from sys import stderr
from threading import Lock

class Progress(object):

  """Basic progress tracker callback."""

  def __init__(self):
    self._data = {}
    self._lock = Lock()

  def __call__(self, hdfs_path, nbytes):
    with self._lock:
      if nbytes >= 0:
          self._data[hdfs_path] = nbytes
      else:
        stderr.write('%s\n' % (sum(self._data.values()), ))

```
Tokens:
Raw