Filesystem Spec

https://filesystem-spec.readthedocs.io/en/latest/
Admin
Filesystem Spec (fsspec) is a project to provide a unified pythonic interface to local, remote and...
Tokens:152,958
Snippets:804
Trust Score:9.6
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install smbprotocol dependency

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/smb.html

Install the required smbprotocol package to enable SMB support.

```bash
$ pip install smbprotocol
# or
# pip install smbprotocol[kerberos]
```

--------------------------------

### Setup Logging Utility

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/utils.html

Configures a logger with a stream handler and specific formatting.

```python
def setup_logging(
    logger: logging.Logger | None = None,
    logger_name: str | None = None,
    level: str = "DEBUG",
    clear: bool = True,
) -> logging.Logger:
    if logger is None and logger_name is None:
        raise ValueError("Provide either logger object or logger name")
    logger = logger or logging.getLogger(logger_name)
    handle = logging.StreamHandler()
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(funcName)s -- %(message)s"
    )
    handle.setFormatter(formatter)
    if clear:
        logger.handlers.clear()
    logger.addHandler(handle)
    logger.setLevel(level)
    return logger
```

--------------------------------

### Install fsspec in development mode

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/developer.rst.txt

Clone the repository and install the package with development, documentation, and testing dependencies.

```bash
git clone https://github.com/fsspec/filesystem_spec
cd filesystem_spec
pip install -e .[dev,doc,test]
```

--------------------------------

### Transaction.start()

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/transaction.html

Starts a transaction on the associated filesystem.

```APIDOC
## Transaction.start()

### Description
Starts a transaction on the associated filesystem, setting the internal transaction state to active.
```

--------------------------------

### Configure filesystem defaults via JSON

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/features.rst.txt

Example of a configuration file content to set default keyword arguments for a specific protocol.

```json
{"file": {"auto_mkdir": true}}
```

--------------------------------

### Initialize CachingFileSystem

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/cached.html

The constructor handles filesystem target configuration, cache storage setup, and mapper initialization.

```python
        super().__init__(**kwargs)
        if fs is None and target_protocol is None:
            raise ValueError(
                "Please provide filesystem instance(fs) or target_protocol"
            )
        if not (fs is None) ^ (target_protocol is None):
            raise ValueError(
                "Both filesystems (fs) and target_protocol may not be both given."
            )
        if cache_storage == "TMP":
            tempdir = tempfile.mkdtemp()
            storage = [tempdir]
            weakref.finalize(self, self._remove_tempdir, tempdir)
        else:
            if isinstance(cache_storage, str):
                storage = [cache_storage]
            else:
                storage = cache_storage
        os.makedirs(storage[-1], exist_ok=True)
        self.storage = storage
        self.kwargs = target_options or {}
        self.cache_check = cache_check
        self.check_files = check_files
        self.expiry = expiry_time
        self.compression = compression

        # Size of cache in bytes. If None then the size is unknown and will be
        # recalculated the next time cache_size() is called. On writes to the
        # cache this is reset to None.
        self._cache_size = None

        if same_names is not None and cache_mapper is not None:
            raise ValueError(
                "Cannot specify both same_names and cache_mapper in "
                "CachingFileSystem.__init__"
            )
        if cache_mapper is not None:
            self._mapper = cache_mapper
        else:
            self._mapper = create_cache_mapper(
                same_names if same_names is not None else False
            )

        self.target_protocol = (
            target_protocol
            if isinstance(target_protocol, str)
            else (fs.protocol if isinstance(fs.protocol, str) else fs.protocol[0])
        )
        self._metadata = CacheMetadata(self.storage)
        self.load_cache()
        self.fs = fs if fs is not None else filesystem(target_protocol, **self.kwargs)

        def _strip_protocol(path):
            # acts as a method, since each instance has a difference target
            return self.fs._strip_protocol(type(self)._strip_protocol(path))

        self._strip_protocol: Callable = _strip_protocol
```

--------------------------------

### Install fsspec

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/index.rst.txt

Standard installation commands for fsspec using pip or conda.

```sh
pip install fsspec
conda install -c conda-forge fsspec
```

--------------------------------

### Get Kwargs from URLs

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Extracts configuration arguments from paths before filesystem instantiation.

```python
@staticmethod
def _get_kwargs_from_urls(path):
    """If kwargs can be encoded in the paths, extract them here

    This should happen before instantiation of the class; incoming paths
    then should be amended to strip the options in methods.

    Examples may look like an sftp path "sftp://user@host:/my/path", where
    the user and host should become kwargs and later get stripped.
    """
    # by default, nothing happens
    return {}
```

--------------------------------

### Mounting Filesystems via Command Line

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/fuse.html

Examples of using the fsspec.fuse module to mount various filesystem types, including chained URLs and custom options.

```bash
python3 -m fsspec.fuse memory /usr/share /tmp/mem
```

```bash
python3 -m fsspec.fuse local /tmp/source /tmp/local \
        -l /tmp/fsspecfuse.log
```

```bash
python3 -m fsspec.fuse 'filecache::zip::file://data.zip' \
        / /tmp/zip \
        -o 'filecache-cache_storage=/tmp/simplecache'
```

```bash
python3 -m fsspec.fuse 'simplecache::ftp://ftp1.at.proftpd.org' \
        /historic/packages/RPMS /tmp/ftp \
        -o 'simplecache-cache_storage=/tmp/simplecache' \
        -o 'simplecache-check_files=false[bool]' \
        -o 'ftp-listings_expiry_time=60[int]' \
        -o 'ftp-username=anonymous' \
        -o 'ftp-password=xieyanbo'
```

--------------------------------

### Mount Filesystem with run()

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/fuse.html

The run function mounts an fsspec filesystem instance to a local directory. It requires fusepy and a system-level FUSE installation.

```python
def run(
    fs,
    path,
    mount_point,
    foreground=True,
    threads=False,
    ready_file=False,
    ops_class=FUSEr,
):
    """Mount stuff in a local directory

    This uses fusepy to make it appear as if a given path on an fsspec
    instance is in fact resident within the local file-system.

    This requires that fusepy by installed, and that FUSE be available on
    the system (typically requiring a package to be installed with
    apt, yum, brew, etc.).
```

--------------------------------

### Directory listing cache structure

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Example of the internal dictionary structure used by DirCache to store directory listings.

```json
{"path0": [
    {"name": "path0/file0",
     "size": 123,
     "type": "file",
     ...
    },
    {"name": "path0/file1",
    },
    ...
    ],
 "path1": [...]
}
```

--------------------------------

### tree(path='/', recursion_limit=2, max_display=25, display_size=False, prefix='', is_last=True, first=True, indent_size=4)

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Returns a string representation of the filesystem structure starting from the specified path.

```APIDOC
## tree(path='/', recursion_limit=2, max_display=25, display_size=False, prefix='', is_last=True, first=True, indent_size=4)

### Description
Return a tree-like structure of the filesystem starting from the given path as a string.

### Parameters
- **path** (str) - Optional - Root path to start traversal from
- **recursion_limit** (int) - Optional - Maximum depth of directory traversal
- **max_display** (int) - Optional - Maximum number of items to display per directory
- **display_size** (bool) - Optional - Whether to display file sizes
- **prefix** (str) - Optional - Current line prefix for visual tree structure
- **is_last** (bool) - Optional - Whether current item is last in its level
- **first** (bool) - Optional - Whether this is the first call
- **indent_size** (int) - Optional - Number of spaces by indent

### Returns
- **str** - A string representing the tree structure.
```

--------------------------------

### Read file blocks with fsspec

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Examples of reading specific byte ranges from a file, optionally using a delimiter to align the read boundaries.

```python
>>> fs.read_block('data/file.csv', 0, 13)  
b'Alice, 100\nBo'
>>> fs.read_block('data/file.csv', 0, 13, delimiter=b'\n')  
b'Alice, 100\nBob, 200\n'
```

--------------------------------

### fsspec.utils.read_block(f, offset, length, delimiter=None, split_before=False)

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Reads a block of bytes from an open file, optionally ensuring the read starts and stops at specified delimiter boundaries.

```APIDOC
## fsspec.utils.read_block

### Description
Read a block of bytes from a file. If a delimiter is provided, the function ensures the read starts and stops at delimiter boundaries following the specified offset and length.

### Parameters
- **f** (IO[bytes]) - Required - Open file object.
- **offset** (int) - Required - Byte offset to start read.
- **length** (int | None) - Required - Number of bytes to read; if None, reads until the end of the file.
- **delimiter** (bytes) - Optional - Ensure reading starts and stops at delimiter bytestring.
- **split_before** (bool) - Optional - Start/stop read before delimiter bytestring.

### Returns
- **bytes** - The block of bytes read from the file.

### Example
```python
from io import BytesIO
f = BytesIO(b'Alice, 100\nBob, 200\nCharlie, 300')
read_block(f, 0, 13, delimiter=b'\n')
# Returns: b'Alice, 100\nBob, 200\n'
```
```

--------------------------------

### Instantiate file-systems

Source: https://filesystem-spec.readthedocs.io/en/latest/usage.html

Demonstrates creating file-system instances using direct class instantiation, registry look-up, and passing backend-specific parameters.

```python
from fsspec.implementations.local import LocalFileSystem

fs = LocalFileSystem()
```

```python
import fsspec

fs = fsspec.filesystem('file')
```

```python
import fsspec

fs = fsspec.filesystem('ftp', host=host, port=port, username=user, password=pw)
```

--------------------------------

### Initialize and Use FSMap

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/mapping.html

Demonstrates creating an FSMap instance and performing basic dictionary-like operations on a filesystem.

```python
>>> fs = FileSystem(**parameters)  # doctest: +SKIP
>>> d = FSMap('my-data/path/', fs)  # doctest: +SKIP
or, more likely
>>> d = fs.get_mapper('my-data/path/')

>>> d['loc1'] = b'Hello World'  # doctest: +SKIP
>>> list(d.keys())  # doctest: +SKIP
['loc1']
>>> d['loc1']  # doctest: +SKIP
b'Hello World'
```

--------------------------------

### Validate engine installation

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Checks if the specified engine is installed, raising an ImportError if not found.

```python
if self.engine == "pyarrow" and find_spec("pyarrow") is None:
    raise ImportError("engine choice `pyarrow` is not installed.")
```

--------------------------------

### Instantiate a filesystem with parameters

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/usage.rst.txt

Pass backend-specific configuration parameters during filesystem creation.

```python
import fsspec

fs = fsspec.filesystem('ftp', host=host, port=port, username=user, password=pw)
```

--------------------------------

### Register a new backend via entry_points

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/developer.rst.txt

Register a custom filesystem protocol in setup.py using setuptools entry_points to make it discoverable by fsspec.

```python
setuptools.setup(
    ...
    entry_points={
        'fsspec.specs': [
            'myfs=myfs.MyFS',
        ],
    },
    ...
)
```

--------------------------------

### Install fsspec with GCS support

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/index.rst.txt

Installation commands for fsspec including optional dependencies for Google Cloud Storage.

```sh
pip install fsspec[gcs]
conda install -c conda-forge gcsfs
```

--------------------------------

### Mounting a filesystem with FUSE

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/fuse.html

This snippet demonstrates the core logic for initializing a filesystem and running the FUSE mount process.

```python
    fs, url_path = url_to_fs(args.url, **kwargs)
    logger.debug("Mounting %s to %s", url_path, str(args.mount_point))
    run(
        fs,
        args.source_path,
        args.mount_point,
        foreground=args.foreground,
        threads=args.threads,
        ready_file=args.ready_file,
        ops_class=fuser,
    )
```

```python
if __name__ == "__main__":
    import sys

    main(sys.argv[1:])
```

--------------------------------

### Get URL info

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html

Retrieves metadata for a URL using HEAD or GET methods without fetching the full content.

```python
async def _info(self, url, **kwargs):
        """Get info of URL

        Tries to access location via HEAD, and then GET methods, but does
        not fetch the data.
        """
```

--------------------------------

### Retrieve file information via HTTP

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html

Attempts to fetch file size information using HEAD or GET requests, falling back to GET if HEAD fails.

```python
        info = {}
        session = await self.set_session()

        for policy in ["head", "get"]:
            try:
                info.update(
                    await _file_info(
                        self.encode_url(url),
                        size_policy=policy,
                        session=session,
                        **self.kwargs,
                        **kwargs,
                    )
                )
                if info.get("size") is not None:
                    break
            except Exception as exc:
                if policy == "get":
                    # If get failed, then raise a FileNotFoundError
                    raise FileNotFoundError(url) from exc
                logger.debug("", exc_info=exc)

        return {"name": url, "size": None, **info, "type": "file"}
```

--------------------------------

### GithubFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/github.html

Initializes the GithubFileSystem instance to connect to a specific GitHub repository.

```APIDOC
## GithubFileSystem(org, repo, sha=None, username=None, token=None, timeout=None, **kwargs)

### Description
Initializes the filesystem interface for a specific GitHub repository. If no SHA is provided, it defaults to the repository's default branch.

### Parameters
- **org** (str) - Required - The GitHub organization or user name.
- **repo** (str) - Required - The repository name.
- **sha** (str) - Optional - The commit SHA, branch, or tag name.
- **username** (str) - Optional - GitHub username for authorized access.
- **token** (str) - Optional - GitHub personal access token for authorized access.
- **timeout** (tuple) - Optional - Connect and read timeouts.
```

--------------------------------

### Get file size

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Retrieves the size of a file or a list of files in bytes.

```python
    def size(self, path):
        """Size in bytes of file"""
        return self.info(path).get("size", None)

    def sizes(self, paths):
        """Size in bytes of each file in a list of paths"""
        return [self.size(p) for p in paths]
```

--------------------------------

### ReferenceFileSystem Initialization Logic

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

The initialization process handles loading reference dictionaries from URLs or direct input, managing template overrides, and configuring remote filesystem connections.

```python
super().__init__(**kwargs)
self.target = target
self.template_overrides = template_overrides
self.simple_templates = simple_templates
self.templates = {}
self.fss = {}
self._dircache = {}
self.max_gap = max_gap
self.max_block = max_block
if isinstance(fo, str):
    dic = dict(
        **(ref_storage_args or target_options or {}), protocol=target_protocol
    )
    ref_fs, fo2 = fsspec.core.url_to_fs(fo, **dic)
    if ".json" not in fo2 and (
        fo.endswith(("parq", "parquet", "/")) or ref_fs.isdir(fo2)
    ):
        # Lazy parquet refs
        logger.info("Open lazy reference dict from URL %s", fo)
        self.references = LazyReferenceMapper(
            fo2,
            fs=ref_fs,
            cache_size=cache_size,
        )
    else:
        # text JSON
        with fsspec.open(fo, "rb", **dic) as f:
            logger.info("Read reference from URL %s", fo)
            text = json.load(f)
        self._process_references(text, template_overrides)
else:
    # dictionaries
    self._process_references(fo, template_overrides)
if isinstance(fs, dict):
    self.fss = {
        k: (
            fsspec.filesystem(k.split(":", 1)[0], **opts)
            if isinstance(opts, dict)
            else opts
        )
        for k, opts in fs.items()
    }
    if None not in self.fss:
        self.fss[None] = filesystem("file")
    return
if fs is not None:
    # single remote FS
    remote_protocol = (
        fs.protocol[0] if isinstance(fs.protocol, tuple) else fs.protocol
    )
    self.fss[remote_protocol] = fs

if remote_protocol is None:
    # get single protocol from any templates
    for ref in self.templates.values():
        if callable(ref):
            ref = ref()
        protocol, _ = fsspec.core.split_protocol(ref)
        if protocol and protocol not in self.fss:
            fs = filesystem(protocol, **(remote_options or {}))
            self.fss[protocol] = fs
if remote_protocol is None:
    # get single protocol from references
    # TODO: warning here, since this can be very expensive?
    for ref in self.references.values():
        if callable(ref):
            ref = ref()
        if isinstance(ref, list) and ref[0]:
            protocol, _ = fsspec.core.split_protocol(ref[0])
            if protocol not in self.fss:
                fs = filesystem(protocol, **(remote_options or {}))
                self.fss[protocol] = fs
                # only use first remote URL
                break

if remote_protocol and remote_protocol not in self.fss:
    fs = filesystem(remote_protocol, **(remote_options or {}))
    self.fss[remote_protocol] = fs
```

--------------------------------

### Initialize FSMap

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Create an FSMap instance by wrapping a FileSystem object or using the get_mapper helper method.

```python
>>> fs = FileSystem(**parameters)  
>>> d = FSMap('my-data/path/', fs)  
or, more likely
>>> d = fs.get_mapper('my-data/path/')
```

--------------------------------

### tree()

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Generates a string representation of the filesystem structure starting from a specified path.

```APIDOC
### tree(path='/', recursion_limit=2, max_display=25, display_size=False, prefix='', is_last=True, first=True, indent_size=4)

Returns a tree-like structure of the filesystem starting from the given path as a string.

#### Parameters
- **path** (str) - Root path to start traversal from
- **recursion_limit** (int) - Maximum depth of directory traversal
- **max_display** (int) - Maximum number of items to display per directory
- **display_size** (bool) - Whether to display file sizes
- **prefix** (str) - Current line prefix for visual tree structure
- **is_last** (bool) - Whether current item is last in its level
- **first** (bool) - Whether this is the first call
- **indent_size** (int) - Number of spaces by indent
```

--------------------------------

### Initialize ReferenceFileSystem Filesystems

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Configures the default filesystem and wraps non-async filesystems to ensure compatibility with asynchronous operations.

```python
self.fss[None] = fs or filesystem("file")  # default one
# Wrap any non-async filesystems to ensure async methods are available below
for k, f in self.fss.items():
    if not f.async_impl:
        self.fss[k] = AsyncFileSystemWrapper(f, asynchronous=self.asynchronous)
    elif self.asynchronous ^ f.asynchronous:
        raise ValueError(
            "Reference-FS's target filesystem must have same value "
            "of asynchronous"
        )
```

--------------------------------

### Retrieve System Umask

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/local.html

Gets the current process umask by temporarily setting it and reverting.

```python
@lru_cache(maxsize=1)
def get_umask(mask: int = 0o666) -> int:
    """Get the current umask.

    Follows https://stackoverflow.com/a/44130549 to get the umask.
    Temporarily sets the umask to the given value, and then resets it to the
    original value.
    """
    value = os.umask(mask)
    os.umask(value)
    return value
```

--------------------------------

### FTPFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/ftp.html

Initializes a new FTPFileSystem instance to connect to a remote FTP server.

```APIDOC
## FTPFileSystem(host, port=21, username=None, password=None, acct=None, block_size=None, tempdir=None, timeout=30, encoding='utf-8', tls=False, **kwargs)

### Description
Initializes the FTP filesystem connection. Authentication is anonymous if username and password are not provided.

### Parameters
- **host** (str) - Required - The remote server name or IP address.
- **port** (int) - Optional - Port to connect with (default 21).
- **username** (str) - Optional - User identifier for authentication.
- **password** (str) - Optional - User password for authentication.
- **acct** (str) - Optional - Account string for servers requiring it.
- **block_size** (int) - Optional - Read-ahead or write buffer size.
- **tempdir** (str) - Optional - Remote directory for temporary files.
- **timeout** (int) - Optional - Connection timeout in seconds (default 30).
- **encoding** (str) - Optional - Encoding for directories and filenames (default 'utf-8').
- **tls** (bool) - Optional - Whether to use FTP-TLS (default False).
```

--------------------------------

### Get file hash

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Returns a SHA256 hash of the file's info to detect changes.

```python
def ukey(self, path):
    """Hash of file properties, to tell if it has changed"""
    return sha256(str(self.info(path)).encode()).hexdigest()
```

--------------------------------

### Transaction Management Methods

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Methods for starting and ending write transactions to defer file operations.

```python
def start_transaction(self):
    """Begin write transaction for deferring files, non-context version"""
    self._intrans = True
    self._transaction = self.transaction_type(self)
    return self.transaction

def end_transaction(self):
    """Finish write transaction, non-context version"""
    self.transaction.complete()
    self._transaction = None
    # The invalid cache must be cleared after the transaction is completed.
    for path in self._invalidated_caches_in_transaction:
        self.invalidate_cache(path)
    self._invalidated_caches_in_transaction.clear()
```

--------------------------------

### ReferenceFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Initializes a new ReferenceFileSystem instance. This class allows mapping paths to specific byte ranges in target files or providing concrete data directly.

```APIDOC
## ReferenceFileSystem(fo, target=None, ...)

### Description
Initializes the ReferenceFileSystem with a dictionary of references or a path to a JSON file containing reference mappings.

### Parameters
- **fo** (dict or str) - Required - The set of references or a path to a JSON file/directory containing references.
- **target** (str) - Optional - Default file target for references where target_url is None.
- **target_options** (dict) - Optional - Extra filesystem options for loading the reference file.
- **remote_protocol** (str) - Optional - The protocol of the filesystem where references are evaluated.
- **remote_options** (dict) - Optional - Arguments for the remote filesystem.
- **fs** (AbstractFileSystem | dict) - Optional - Directly provide a filesystem instance or a mapping of protocols to filesystem configurations.
```

--------------------------------

### Parse Content-Range Header

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html

Extracts start, end, and total size from the HTTP Content-Range header.

```python
def _parse_content_range(self, headers):
    """Parse the Content-Range header"""
    s = headers.get("Content-Range", "")
    m = re.match(r"bytes (\d+-\d+|\*)/(\d+|\*)", s)
    if not m:
        return None, None, None

    if m[1] == "*":
        start = end = None
    else:
        start, end = [int(x) for x in m[1].split("-")]
    total = None if m[2] == "*" else int(m[2])
    return start, end, total
```

--------------------------------

### SMBFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/smb.html

Initializes the SMBFileSystem connection to a remote server.

```APIDOC
## SMBFileSystem.__init__

### Description
Initializes a new instance of the SMBFileSystem to interact with a remote SMB server.

### Parameters
- **host** (str) - Required - The remote server name or IP address.
- **port** (int or None) - Optional - Port to connect with (usually 445 or 139).
- **username** (str or None) - Optional - Username for authentication.
- **password** (str or None) - Optional - Password for the specified user.
- **timeout** (int) - Optional - Connection timeout in seconds (default 60).
- **encrypt** (bool) - Optional - Whether to force encryption.
- **share_access** (str or None) - Optional - Default access applied to file open operations.
- **auto_mkdir** (bool) - Optional - Whether to automatically create directories.
```

--------------------------------

### Verify filesystem configuration overrides

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/features.rst.txt

Demonstrates how explicit keyword arguments override default configuration settings.

```python
import fsspec
fs = fsspec.filesystem("file")
assert fs.auto_mkdir == True
fs = fsspec.filesystem("file", auto_mkdir=False)
assert fs.auto_mkdir == False
```

--------------------------------

### Handle attribute access

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Intercepts attribute access to trigger setup for specific metadata-related attributes.

```python
def __getattr__(self, item):
    if item in ("_items", "record_size", "zmetadata"):
        self.setup()
        # avoid possible recursion if setup fails somehow
        return self.__dict__[item]
    raise AttributeError(item)
```

--------------------------------

### Instantiate a filesystem

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/registry.html

Creates an instance of a filesystem class based on the provided protocol and storage options.

```python
def filesystem(protocol, **storage_options):
    """Instantiate filesystems for given protocol and arguments

    ``storage_options`` are specific to the protocol being chosen, and are
    passed directly to the class.
    """
    if protocol == "arrow_hdfs":
        warnings.warn(
            "The 'arrow_hdfs' protocol has been deprecated and will be "
            "removed in the future. Specify it as 'hdfs'.",
            DeprecationWarning,
        )

    cls = get_filesystem_class(protocol)
    return cls(**storage_options)
```

--------------------------------

### Retrieve File Information

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html

Call HEAD or GET on the server to retrieve file metadata like size.

```python
async def _file_info(url, session, size_policy="head", **kwargs):
    """Call HEAD on the server to get details about the file (size/checksum etc.)

    Default operation is to explicitly allow redirects and use encoding
    'identity' (no compression) to get the true size of the target.
    """
    logger.debug("Retrieve file size for %s", url)
    kwargs = kwargs.copy()
    ar = kwargs.pop("allow_redirects", True)
    head = kwargs.get("headers", {}).copy()
    head["Accept-Encoding"] = "identity"
    kwargs["headers"] = head

    info = {}
    if size_policy == "head":
        r = await session.head(url, allow_redirects=ar, **kwargs)
    elif size_policy == "get":
        r = await session.get(url, allow_redirects=ar, **kwargs)
    else:
        raise TypeError(f'size_policy must be "head" or "get", got {size_policy}')
    async with r:
        r.raise_for_status()
```

--------------------------------

### Check file existence via HTTP

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html

Verifies if a remote path exists by performing a GET request.

```python
async def _exists(self, path, strict=False, **kwargs):
        kw = self.kwargs.copy()
        kw.update(kwargs)
        try:
            logger.debug(path)
            session = await self.set_session()
            r = await session.get(self.encode_url(path), **kw)
            async with r:
                if strict:
                    self._raise_not_found_for_status(r, path)
                return r.status < 400
        except FileNotFoundError:
            return False
        except aiohttp.ClientError:
            if strict:
                raise
            return False
```

--------------------------------

### ZipFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/zip.html

Initializes the ZipFileSystem instance to interact with a ZIP archive.

```APIDOC
## ZipFileSystem(fo, mode='r', target_protocol=None, target_options=None, compression=zipfile.ZIP_STORED, allowZip64=True, compresslevel=None, **kwargs)

### Description
Initializes a new ZipFileSystem instance. The instance keeps the file object open while it lives.

### Parameters
- **fo** (str or file-like) - Required - Contains the ZIP archive. If a string, it is fetched using fsspec.open_files.
- **mode** (str) - Optional - Access mode: 'r', 'w', or 'a'. Defaults to 'r'.
- **target_protocol** (str) - Optional - Protocol to override if fo is a string.
- **target_options** (dict) - Optional - Kwargs passed to the target filesystem if fo is a string.
- **compression** (int) - Optional - Compression method passed to ZipFile.
- **allowZip64** (bool) - Optional - Whether to allow Zip64 extensions.
- **compresslevel** (int) - Optional - Compression level passed to ZipFile.
```

--------------------------------

### Get Current Filesystem Instance

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Returns the most recently instantiated FileSystem, or creates a new one if none exists.

```python
@classmethod
def current(cls):
    """Return the most recently instantiated FileSystem

    If no instance has been created, then create one with defaults
    """
    if cls._latest in cls._cache:
        return cls._cache[cls._latest]
    return cls()
```

--------------------------------

### GistFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/gist.html

Initializes the GistFileSystem to access a specific GitHub Gist.

```APIDOC
## GistFileSystem(gist_id, filenames=None, sha=None, username=None, token=None, timeout=None, **kwargs)

### Description
Initializes a new instance of the GistFileSystem to provide read-only access to files in a single GitHub Gist.

### Parameters
- **gist_id** (str) - Required - The ID of the gist to access (the long hex value from the URL).
- **filenames** (list[str]) - Optional - If provided, only represent these specific files.
- **sha** (str) - Optional - The specific revision of the gist to fetch. Defaults to the latest revision.
- **username** (str) - Optional - GitHub username for authentication.
- **token** (str) - Optional - GitHub personal access token (required if username is provided).
- **timeout** (float or tuple) - Optional - Connect and read timeouts for requests (default 60s).
- **kwargs** (dict) - Optional - Additional arguments passed to requests.get when fetching metadata or reading files.
```

--------------------------------

### CachingFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/cached.html

Initializes a new CachingFileSystem instance to manage local caching for a target filesystem.

```APIDOC
## CachingFileSystem.__init__

### Description
Initializes the CachingFileSystem with storage locations, cache policies, and the target filesystem.

### Parameters
- **target_protocol** (str) - Optional - Target filesystem protocol.
- **cache_storage** (str or list(str)) - Required - Location to store files. Use 'TMP' for temporary storage.
- **cache_check** (int) - Optional - Number of seconds between reload of cache metadata.
- **check_files** (bool) - Optional - Whether to verify remote file UID against stored file.
- **expiry_time** (int) - Optional - Time in seconds after which a local copy is considered expired.
- **target_options** (dict) - Optional - Options passed to the target filesystem instantiation.
- **fs** (filesystem instance) - Optional - The target filesystem instance to wrap.
- **same_names** (bool) - Optional - If true, uses BasenameCacheMapper instead of HashCacheMapper.
- **compression** (str) - Optional - Decompression strategy for downloads.
- **cache_mapper** (AbstractCacheMapper) - Optional - Custom mapper for filename mapping.
```

--------------------------------

### Get chunk sizes for a field

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Determines the number of chunks along each axis for a specific field using zmetadata.

```python
    def _get_chunk_sizes(self, field):
        """The number of chunks along each axis for a given field"""
        if field not in self.chunk_sizes:
            zarray = self.zmetadata[f"{field}/.zarray"]
            size_ratio = [
                math.ceil(s / c) for s, c in zip(zarray["shape"], zarray["chunks"])
            ]
            self.chunk_sizes[field] = size_ratio or [1]
        return self.chunk_sizes[field]
```

--------------------------------

### Send API request to DBFS

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/dbfs.html

Helper method to execute GET or POST requests against the DBFS REST API.

```python
    def _send_to_api(self, method, endpoint, json):
        """
        Send the given json to the DBFS API
        using a get or post request (specified by the argument `method`).

        Parameters
        ----------
        method: str
            Which http method to use for communication; "get" or "post".
        endpoint: str
            Where to send the request to (last part of the API URL)
        json: dict
            Dictionary of information to send
        """
        if method == "post":
            session_call = self.session.post
        elif method == "get":
            session_call = self.session.get
        else:
            raise ValueError(f"Do not understand method {method}")

        url = urllib.parse.urljoin(f"https://{self.instance}/api/2.0/dbfs/", endpoint)

        r = session_call(url, json=json)
```

--------------------------------

### Instantiate a filesystem via registry

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/usage.rst.txt

Use the fsspec registry to look up and instantiate a filesystem by its protocol name.

```python
import fsspec

fs = fsspec.filesystem('file')
```

--------------------------------

### LocalFileSystem

Source: https://filesystem-spec.readthedocs.io/en/latest/api.html

Initializes an interface for interacting with local storage.

```APIDOC
## LocalFileSystem(auto_mkdir=False, **kwargs)

### Description
Creates and configures a local filesystem instance.

### Parameters
- **auto_mkdir** (bool) - Optional - Whether to create the directory containing a file when opening it.
- **use_listings_cache** (bool) - Optional - Passed to DirCache if supported.
- **listings_expiry_time** (float) - Optional - Expiry time for directory listing cache.
- **max_paths** (int) - Optional - Maximum paths for directory listing cache.
- **skip_instance_cache** (bool) - Optional - Force creating a new instance even if a matching one exists.
```

--------------------------------

### cat_ranges(paths, starts, ends, max_gap=None, on_error='return', **kwargs)

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Fetches the contents of specific byte ranges from one or more files.

```APIDOC
## cat_ranges(paths, starts, ends, max_gap=None, on_error='return', **kwargs)

### Description
Get the contents of byte ranges from one or more files.

### Parameters
- **paths** (list) - Required - A list of filepaths on this filesystem.
- **starts** (int or list) - Required - Byte limits of the read start. If using a single int, the same value will be used for all files.
- **ends** (int or list) - Required - Byte limits of the read end. If using a single int, the same value will be used for all files.
```

--------------------------------

### filesystem(protocol, **storage_options)

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/registry.html

Instantiates a filesystem class for the given protocol, passing any additional storage_options directly to the class constructor.

```APIDOC
## filesystem(protocol, **storage_options)

### Description
Instantiate filesystems for a given protocol and arguments. The storage_options are specific to the protocol being chosen and are passed directly to the class constructor.

### Parameters
- **protocol** (str) - Required - The protocol identifier for the filesystem (e.g., 's3', 'hdfs').
- **storage_options** (dict) - Optional - Additional keyword arguments passed to the filesystem class constructor.
```

--------------------------------

### Chain Filesystem Protocols for Zip Files

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/features.rst.txt

Examples of chaining protocols to read and cache zip files from remote storage.

```python
of = fsspec.open_files('zip://*.csv::simplecache::gcs://bucket/afile.zip',
                       simplecache={'cache_storage': '/stored/zip/files'},
                       gcs={'project': 'my-project'})
```

```python
of = fsspec.open_files('simplecache::zip://*.csv::gcs://bucket/afile.zip',
                       simplecache={'cache_storage': '/stored/csv/files'},
                       gcs={'project': 'my-project'})
```

--------------------------------

### DatabricksFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/dbfs.html

Initializes a new DatabricksFileSystem instance to interact with a Databricks cluster.

```APIDOC
## DatabricksFileSystem(instance, token, **kwargs)

### Description
Creates a new DatabricksFileSystem instance. This can be used both inside and outside of a Databricks cluster.

### Parameters
- **instance** (str) - Required - The instance URL of the Databricks cluster (e.g., adb-<number>.<digits>.azuredatabricks.net).
- **token** (str) - Required - The personal access token for authentication.
```

--------------------------------

### Open files with fsspec

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/core.html

Examples of opening local and remote files using the open function, including support for compression.

```python
>>> openfile = open('2015-01-01.csv')  # doctest: +SKIP
>>> openfile = open(
...     's3://bucket/2015-01-01.csv.gz', compression='gzip'
... )  # doctest: +SKIP
>>> with openfile as f:
...     df = pd.read_csv(f)  # doctest: +SKIP
```

--------------------------------

### Instantiate a LocalFileSystem directly

Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/usage.rst.txt

Directly import and instantiate a specific filesystem class.

```python
from fsspec.implementations.local import LocalFileSystem

fs = LocalFileSystem()
```

--------------------------------

### Initialize ReferenceFileSystem

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html

Constructor for the ReferenceFileSystem class, which accepts reference dictionaries or paths to JSON files.

```python
class ReferenceFileSystem(AsyncFileSystem):
    """View byte ranges of some other file as a file system
    Initial version: single file system target, which must support
    async, and must allow start and end args in _cat_file. Later versions
    may allow multiple arbitrary URLs for the targets.
    This FileSystem is read-only. It is designed to be used with async
    targets (for now). We do not get original file details from the target FS.
    Configuration is by passing a dict of references at init, or a URL to
    a JSON file containing the same; this dict
    can also contain concrete data for some set of paths.
    Reference dict format:
    {path0: bytes_data, path1: (target_url, offset, size)}
    https://github.com/fsspec/kerchunk/blob/main/README.md
    """

    protocol = "reference"
    cachable = False

    def __init__(
        self,
        fo,
        target=None,
        ref_storage_args=None,
        target_protocol=None,
        target_options=None,
        remote_protocol=None,
        remote_options=None,
        fs=None,
        template_overrides=None,
        simple_templates=True,
        max_gap=64_000,
        max_block=256_000_000,
        cache_size=128,
        **kwargs,
    ):
        """
        Parameters
        ----------
        fo : dict or str
            The set of references to use for this instance, with a structure as above.
            If str referencing a JSON file, will use fsspec.open, in conjunction
            with target_options and target_protocol to open and parse JSON at this
            location. If a directory, then assume references are a set of parquet
            files to be loaded lazily.
        target : str
            For any references having target_url as None, this is the default file
            target to use
        ref_storage_args : dict
            If references is a str, use these kwargs for loading the JSON file.
            Deprecated: use target_options instead.
        target_protocol : str
            Used for loading the reference file, if it is a path. If None, protocol
            will be derived from the given path
        target_options : dict
            Extra FS options for loading the reference file ``fo``, if given as a path
        remote_protocol : str
            The protocol of the filesystem on which the references will be evaluated
            (unless fs is provided). If not given, will be derived from the first
            URL that has a protocol in the templates or in the references, in that
            order.
        remote_options : dict
            kwargs to go with remote_protocol
        fs : AbstractFileSystem | dict(str, (AbstractFileSystem | dict))
            Directly provide a file system(s):
                - a single filesystem instance
                - a dict of protocol:filesystem, where each value is either a filesystem
                  instance, or a dict of kwargs that can be used to create in
                  instance for the given protocol
        """
```

--------------------------------

### GitFileSystem.__init__

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/git.html

Initializes the GitFileSystem instance to interact with a local git repository.

```APIDOC
## GitFileSystem(path=None, fo=None, ref=None, **kwargs)

### Description
Initializes a new GitFileSystem instance. The filesystem can be pointed to a local repository path and a default reference (hash, tag, or branch).

### Parameters
- **path** (str) - Optional - Local location of the repo. Defaults to current directory.
- **fo** (str) - Optional - Alternative path parameter, takes precedence if provided.
- **ref** (str) - Optional - Default reference (hash, tag, or branch) to work with. Defaults to 'master'.
```

--------------------------------

### get(rpath, lpath, recursive=False, callback=DEFAULT_CALLBACK, maxdepth=None, **kwargs)

Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html

Copies a specific file or tree of files from a remote path to a local path.

```APIDOC
## get(rpath, lpath, recursive=False, callback=DEFAULT_CALLBACK, maxdepth=None, **kwargs)

### Description
Copies a specific file or tree of files (if recursive=True) from remote to local. If lpath ends with a "/", it will be assumed to be a directory, and target files will go within. Can submit a list of paths, which may be glob-patterns and will be expanded.

### Parameters
- **rpath** (str or list) - Required - Remote path(s) to copy from.
- **lpath** (str or list) - Required - Local path(s) to copy to.
- **recursive** (bool) - Optional - Whether to copy directories recursively.
- **callback** (object) - Optional - Callback object for progress tracking.
- **maxdepth** (int) - Optional - Maximum depth for recursive operations.
```
Tokens:
Raw