### Install smbprotocol Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Shows the command to install the necessary `smbprotocol` library for SMB share access with fsspec. ```bash pip install smbprotocol ``` -------------------------------- ### Get Available Protocols Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Returns a list of supported file system protocols. Note that some protocols may require additional packages to be installed. ```python fsspec.available_protocols() ``` -------------------------------- ### Installing fsspec in development mode Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/developer.rst.txt Install fsspec and its development dependencies from a cloned repository. This setup is suitable for contributing to the fsspec library. ```bash git clone https://github.com/fsspec/filesystem_spec cd filesystem_spec pip install -e .[dev,doc,test] ``` -------------------------------- ### Instantiate FileSystem via Registry Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/usage.rst.txt Use the fsspec registry to get a file system instance by protocol name. ```python import fsspec fs = fsspec.filesystem('file') ``` -------------------------------- ### Setup Logging Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/utils.html Configures a logger with a stream handler and a specific format. Useful for setting up application-wide logging. ```python def setup_logging( logger: logging.Logger | None = None, logger_name: str | None = None, level: str = "DEBUG", clear: bool = True, ) -> logging.Logger: if logger is None and logger_name is None: raise ValueError("Provide either logger object or logger name") logger = logger or logging.getLogger(logger_name) handle = logging.StreamHandler() formatter = logging.Formatter( "%(asctime)s - %(name)s - %(levelname)s - %(funcName)s -- %(message)s" ) handle.setFormatter(formatter) if clear: logger.handlers.clear() logger.addHandler(handle) logger.setLevel(level) return logger ``` -------------------------------- ### get Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Copies file(s) to the local filesystem. ```APIDOC ## get ### Description Copy file(s) to local. ### Method Not specified (likely part of a class method) ### Endpoint Not applicable (method call) ### Parameters * **rpath** (str or list[str]) - The remote path(s) to copy. Can be a single path, a list of paths, or glob patterns. * **lpath** (str) - The local path to copy to. If it ends with '/', it's treated as a directory. * **recursive** (bool, optional) - If True, copies the entire directory tree. Defaults to False. * **callback** (object, optional) - A callback object for progress reporting. * **maxdepth** (int or None, optional) - Maximum depth for recursive copies. * **kwargs** - Additional keyword arguments. ### Response Not specified. Calls `get_file` for each source. ``` -------------------------------- ### FSMap Initialization and Basic Usage Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/mapping.html Demonstrates how to initialize an FSMap and perform basic operations like setting and retrieving values. Assumes a FileSystem instance is already created. ```python fs = FileSystem(**parameters) # doctest: +SKIP d = FSMap('my-data/path/', fs) # doctest: +SKIP or, more likely d = fs.get_mapper('my-data/path/') d['loc1'] = b'Hello World' # doctest: +SKIP list(d.keys()) # doctest: +SKIP ['loc1'] d['loc1'] # doctest: +SKIP b'Hello World' ``` -------------------------------- ### Get Available Compressions Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Returns a list of compression algorithms implemented by fsspec. No setup is required. ```python fsspec.available_compressions() ``` -------------------------------- ### Create FSMap Instance Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Demonstrates how to create an FSMap instance, which wraps a FileSystem instance to provide dictionary-like access to files. This is useful for treating file system paths as dictionary keys. ```python fs = FileSystem(**parameters) >>> d = FSMap('my-data/path/', fs) or, more likely >>> d = fs.get_mapper('my-data/path/') ``` -------------------------------- ### Instantiate FileSystem with Parameters Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/usage.rst.txt Instantiate a file system like FTP with specific connection parameters. ```python import fsspec fs = fsspec.filesystem('ftp', host=host, port=port, username=user, password=pw) ``` -------------------------------- ### Read a block with offset and newline delimiter Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Reads a block of bytes starting from a specific offset, using newline as a delimiter. This example demonstrates reading a middle section of the data. ```python read_block(f, 10, 10, delimiter=b'\n') ``` -------------------------------- ### Install fsspec with GCS support Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/index.rst.txt Install fsspec with specific filesystem implementations, such as GCS, using optional pip syntax or by installing the dedicated package. ```sh pip install fsspec[gcs] conda install -c conda-forge gcsfs ``` -------------------------------- ### Instantiate FileSystem with default configuration Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/features.rst.txt Demonstrates how to instantiate a file system and access configuration values that were set by a config file. The auto_mkdir attribute is true by default. ```python import fsspec fs = fsspec.filesystem("file") assert fs.auto_mkdir == True ``` -------------------------------- ### DirFs: File Download (get) Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/dirfs.html The `get` method downloads data from a remote path to a local path. It prepends the DirFs base path to the remote path before calling the underlying filesystem's `get` method. ```python def get(self, rpath, *args, **kwargs): return self.fs.get(self._join(rpath), *args, **kwargs) ``` -------------------------------- ### Initialize HadoopFileSystem Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Create a HadoopFileSystem instance to connect to HDFS using pyarrow.fs.HadoopFileSystem. Configuration options for host, port, user, and replication are available. ```python fs = HadoopFileSystem(host='default', port=0, user=None, kerb_ticket=None, replication=3, extra_conf=None) ``` -------------------------------- ### Registering a new fsspec backend with setuptools Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/developer.rst.txt Register a new filesystem protocol by adding an entry point to your setup.py file under the 'fsspec.specs' key. ```python setuptools.setup( ... entry_points={ 'fsspec.specs': [ 'myfs=myfs.MyFS', ], }, ... ) ``` -------------------------------- ### Install fsspec Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/index.rst.txt Install the fsspec package using pip or conda. No direct dependencies are required for the base package. ```sh pip install fsspec conda install -c conda-forge fsspec ``` -------------------------------- ### Make Directories (MemoryFileSystem) Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/memory.html Creates a directory and any necessary parent directories. If exist_ok is True, it will not raise an error if the directory already exists. ```python fs = MemoryFileSystem() fs.makedirs('/a/b/c') fs.makedirs('/a/b/c', exist_ok=True) ``` -------------------------------- ### Instantiate Filesystem Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Creates a filesystem instance for a specified protocol. Protocol-specific storage options can be passed as keyword arguments. ```python fsspec.filesystem(_protocol_, **storage_options) ``` -------------------------------- ### TqdmCallback Example Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Use TqdmCallback to display a progress bar during file uploads. This example shows a basic upload with a TqdmCallback. ```python import fsspec from fsspec.callbacks import TqdmCallback fs = fsspec.filesystem("memory") path2distant_data = "/your-path" fs.upload( ".", path2distant_data, recursive=True, callback=TqdmCallback(), ) ``` -------------------------------- ### current Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Returns the most recently instantiated FileSystem. If no instance exists, it creates one with default settings. ```APIDOC ## current ### Description Return the most recently instantiated FileSystem. If no instance has been created, then create one with defaults. ``` -------------------------------- ### Get File Size via HTTP Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html A synchronous wrapper to get the size of a file at a given URL using HTTP. It utilizes the _file_info function internally. ```python async def _file_size(url, session=None, *args, **kwargs): if session is None: session = await get_client() info = await _file_info(url, session=session, *args, **kwargs) return info.get("size") file_size = sync_wrapper(_file_size) ``` -------------------------------- ### Demonstrate Local File System auto_mkdir Source: https://filesystem-spec.readthedocs.io/en/latest/features.html Instantiate the 'file' filesystem and assert its auto_mkdir behavior, first with default configuration and then with an explicit override. ```python import fsspec fs = fsspec.filesystem("file") assert fs.auto_mkdir == True fs = fsspec.filesystem("file", auto_mkdir=False) assert fs.auto_mkdir == False ``` -------------------------------- ### Get Operation for Files Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Handles the `get` operation, which can retrieve single files or recursively download directories. It uses `cat` to fetch data and `fs.pipe_file` to write it to the local filesystem. ```python def get(self, rpath, lpath, recursive=False, **kwargs): if recursive: # trigger directory build self.ls("") rpath = self.expand_path(rpath, recursive=recursive) fs = fsspec.filesystem("file", auto_mkdir=True) targets = other_paths(rpath, lpath) if recursive: data = self.cat([r for r in rpath if not self.isdir(r)]) else: data = self.cat(rpath) for remote, local in zip(rpath, targets): if remote in data: fs.pipe_file(local, data[remote]) ``` -------------------------------- ### Get Range of Data from URL Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Fetches a specific byte range from a URL using an HTTP GET request with a 'Range' header. Optionally writes the fetched data to a file. ```python async def get_range(session, url, start, end, file=None, **kwargs): # explicit get a range when we know it must be safe kwargs = kwargs.copy() headers = kwargs.pop("headers", {}).copy() headers["Range"] = f"bytes={start}-{end - 1}" r = await session.get(url, headers=headers, **kwargs) r.raise_for_status() async with r: out = await r.read() if file: with open(file, "r+b") as f: # noqa: ASYNC230 f.seek(start) f.write(out) else: return out ``` -------------------------------- ### ReferenceFile _fetch_range method Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Fetches a specific range of bytes from the underlying file. It adjusts the start and end points based on the file's defined start and end offsets. ```python def _fetch_range(self, start, end): start = start + self.start end = min(end + self.start, self.end) self.f.seek(start) return self.f.read(end - start) ``` -------------------------------- ### Instantiating Filesystems with Protocol and Options Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/registry.html This function creates an instance of a filesystem for a given protocol and passes any additional storage options directly to the filesystem's constructor. It also includes a deprecation warning for the 'arrow_hdfs' protocol. ```python def filesystem(protocol, **storage_options): """Instantiate filesystems for given protocol and arguments ``storage_options`` are specific to the protocol being chosen, and are passed directly to the class. """ if protocol == "arrow_hdfs": warnings.warn( "The 'arrow_hdfs' protocol has been deprecated and will be " "removed in the future. Specify it as 'hdfs'.", DeprecationWarning, ) ``` -------------------------------- ### Get HTTP File Metadata Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Retrieves metadata for a URL using HEAD or GET requests without fetching the full data. Handles cases where size information might be missing. ```python async def _info(self, url, **kwargs): """Get info of URL Tries to access location via HEAD, and then GET methods, but does not fetch the data. It is possible that the server does not supply any size information, in which case size will be given as None (and certain operations on the corresponding file will not work). """ info = {} session = await self.set_session() for policy in ["head", "get"]: try: info.update( await _file_info( self.encode_url(url), size_policy=policy, session=session, **self.kwargs, **kwargs, ) ) if info.get("size") is not None: ``` -------------------------------- ### Instantiate FileSystem overriding default configuration Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/features.rst.txt Shows how to instantiate a file system and explicitly override default configuration values by passing keyword arguments. Here, auto_mkdir is set to False. ```python fs = fsspec.filesystem("file", auto_mkdir=False) assert fs.auto_mkdir == False ``` -------------------------------- ### Asynchronous Get File Operation Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Handles the asynchronous retrieval of a file from the reference filesystem to a local path. It first checks if the path is a directory and then uses `_cat_file` to get the data, writing it to the local file. ```python async def _get_file(self, rpath, lpath, **kwargs): if self.isdir(rpath): return os.makedirs(lpath, exist_ok=True) data = await self._cat_file(rpath) with open(lpath, "wb") as f: f.write(data) ``` -------------------------------- ### FileSelector Initialization and Basic Usage Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/gui.html Instantiates a FileSelector and demonstrates how to open a selected file in binary read mode within a notebook environment. The example shows the typical workflow of selecting a file and then using the `open_file` context manager. ```python sel = FileSelector() sel ``` ```python with sel.open_file('rb') as f: out = f.read() ``` -------------------------------- ### FSMap Class Methods Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/mapping.html Demonstrates various methods of the FSMap class for interacting with file system data as a key-value store. Includes methods for retrieving, setting, deleting, and checking for the existence of keys. ```python def _str_to_key(self, s): """Strip path of to leave key name""" return s[len(self.root) :].lstrip("/") def __getitem__(self, key, default=None): """Retrieve data""" k = self._key_to_str(key) try: result = self.fs.cat(k) except self.missing_exceptions as exc: if default is not None: return default raise KeyError(key) from exc return result [docs] def pop(self, key, default=None): """Pop data""" result = self.__getitem__(key, default) try: del self[key] except KeyError: pass return result def __setitem__(self, key, value): """Store value in key""" key = self._key_to_str(key) self.fs.mkdirs(self.fs._parent(key), exist_ok=True) self.fs.pipe_file(key, maybe_convert(value)) def __iter__(self): return (self._str_to_key(x) for x in self.fs.find(self.root)) def __len__(self): return len(self.fs.find(self.root)) def __delitem__(self, key): """Remove key""" try: self.fs.rm(self._key_to_str(key)) except Exception as exc: raise KeyError from exc def __contains__(self, key): """Does key exist in mapping?""" path = self._key_to_str(key) return self.fs.isfile(path) def __reduce__(self): return FSMap, (self.root, self.fs, False, False, self.missing_exceptions) ``` -------------------------------- ### Get File Information via HTTP Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Retrieves metadata for a file located at a given URL using an HTTP GET request. Handles potential redirects and extracts relevant headers like ETag and Content-MD5. ```python async def _file_info(url, session=None, *args, **kwargs): if session is None: session = await get_client() # HEAD request to get info without downloading the body try: r = await session.head(url, allow_redirects=True, *args, **kwargs) r.raise_for_status() except HTTPError as e: # If HEAD fails, try GET, but only if it's not a redirect error if e.response.status_code != 405: raise r = await session.get(url, allow_redirects=True, *args, **kwargs) r.raise_for_status() info = { "name": url.split("/")[-1], "size": int(r.headers.get("content-length", 0)), "type": "file", } # Check for partial content support if "bytes" not in r.headers.get("accept-ranges", "none"): # the lack of "Accept-Ranges" does not always indicate they would fail info["partial"] = False info["url"] = str(r.url) for checksum_field in ["ETag", "Content-MD5", "Digest", "Last-Modified"]: if r.headers.get(checksum_field): info[checksum_field] = r.headers[checksum_field] return info ``` -------------------------------- ### HTTP GET Request and Content Parsing Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Handles GET requests for URLs, parses HTML content to extract links, and distinguishes between binary and text files. It supports simple link extraction and more complex regex-based parsing. ```python async with session.get(self.encode_url(url), **self.kwargs) as r: self._raise_not_found_for_status(r, url) if "Content-Type" in r.headers: mimetype = r.headers["Content-Type"].partition(";")[0] else: mimetype = None if mimetype in ("text/html", None): try: text = await r.text(errors="ignore") if self.simple_links: links = ex2.findall(text) + [u[2] for u in ex.findall(text)] else: links = [u[2] for u in ex.findall(text)] except UnicodeDecodeError: links = [] # binary, not HTML else: links = [] out = set() parts = urlparse(url) for l in links: if isinstance(l, tuple): l = l[1] if l.startswith("/") and len(l) > 1: # absolute URL on this server l = f"{parts.scheme}://{parts.netloc}{l}" if l.startswith("http"): if self.same_schema and l.startswith(url.rstrip("/") + "/"): out.add(l) elif l.replace("https", "http").startswith( url.replace("https", "http").rstrip("/") + "/" ): # allowed to cross http <-> https out.add(l) else: if l not in ["..", "../"]: # Ignore FTP-like "parent" out.add("/”.join([url.rstrip("/"), l.lstrip("/")])) if not out and url.endswith("/"): out = await self._ls_real(url.rstrip("/"), detail=False) if detail: return [ { "name": u, "size": None, "type": "directory" if u.endswith("/") else "file", } for u in out ] else: return sorted(out) ``` -------------------------------- ### ReferenceFileSystem Initialization Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Initializes the ReferenceFileSystem with various configuration options for managing references, target filesystems, and request merging. ```python def __init__( self, fo, target=None, ref_storage_args=None, target_protocol=None, target_options=None, remote_protocol=None, remote_options=None, fs=None, template_overrides=None, simple_templates=True, max_gap=64_000, max_block=256_000_000, cache_size=128, **kwargs, ): """ Parameters ---------- fo : dict or str The set of references to use for this instance, with a structure as above. If str referencing a JSON file, will use fsspec.open, in conjunction with target_options and target_protocol to open and parse JSON at this location. If a directory, then assume references are a set of parquet files to be loaded lazily. target : str For any references having target_url as None, this is the default file target to use ref_storage_args : dict If references is a str, use these kwargs for loading the JSON file. Deprecated: use target_options instead. target_protocol : str Used for loading the reference file, if it is a path. If None, protocol will be derived from the given path target_options : dict Extra FS options for loading the reference file ``fo``, if given as a path remote_protocol : str The protocol of the filesystem on which the references will be evaluated (unless fs is provided). If not given, will be derived from the first URL that has a protocol in the templates or in the references, in that order. remote_options : dict kwargs to go with remote_protocol fs : AbstractFileSystem | dict(str, (AbstractFileSystem | dict)) Directly provide a file system(s): - a single filesystem instance - a dict of protocol:filesystem, where each value is either a filesystem instance, or a dict of kwargs that can be used to create in instance for the given protocol If this is given, remote_options and remote_protocol are ignored. template_overrides : dict Swap out any templates in the references file with these - useful for testing. simple_templates: bool Whether templates can be processed with simple replace (True) or if jinja is needed (False, much slower). All reference sets produced by ``kerchunk`` are simple in this sense, but the spec allows for complex. max_gap, max_block: int For merging multiple concurrent requests to the same remote file. Neighboring byte ranges will only be merged when their inter-range gap is <= ``max_gap``. Default is 64KB. Set to 0 to only merge when it requires no extra bytes. Pass a negative number to disable merging, appropriate for local target files. Neighboring byte ranges will only be merged when the size of the aggregated range is <= ``max_block``. Default is 256MB. cache_size : int Maximum size of LRU cache, where cache_size*record_size denotes the total number of references that can be loaded in memory at once. ``` -------------------------------- ### Retrieve File Information via HTTP HEAD or GET Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Fetches file metadata such as size and MIME type by making an HTTP HEAD or GET request. It prioritizes HEAD requests and handles potential content encoding issues. ```python async def _file_info(url, session, size_policy="head", **kwargs): """Call HEAD on the server to get details about the file (size/checksum etc.) Default operation is to explicitly allow redirects and use encoding 'identity' (no compression) to get the true size of the target. """ logger.debug("Retrieve file size for %s", url) kwargs = kwargs.copy() ar = kwargs.pop("allow_redirects", True) head = kwargs.get("headers", {}).copy() head["Accept-Encoding"] = "identity" kwargs["headers"] = head info = {} if size_policy == "head": r = await session.head(url, allow_redirects=ar, **kwargs) elif size_policy == "get": r = await session.get(url, allow_redirects=ar, **kwargs) else: raise TypeError(f'size_policy must be "head" or "get", got {size_policy}') async with r: r.raise_for_status() if "Content-Length" in r.headers: # Some servers may choose to ignore Accept-Encoding and return # compressed content, in which case the returned size is unreliable. if "Content-Encoding" not in r.headers or r.headers["Content-Encoding"] in [ "identity", "", ]: info["size"] = int(r.headers["Content-Length"]) elif "Content-Range" in r.headers: info["size"] = int(r.headers["Content-Range"].split("/")[1]) if "Content-Type" in r.headers: info["mimetype"] = r.headers["Content-Type"].partition(";")[0] if r.headers.get("Accept-Ranges") == "none": # Some servers may explicitly discourage partial content requests, but ``` -------------------------------- ### Reference Filesystem Initialization with Multiple Filesystems Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Initializes the reference filesystem with a dictionary of other filesystems. It ensures that a default 'file' filesystem is available. ```python if isinstance(fs, dict): self.fss = { k: ( fsspec.filesystem(k.split(":", 1)[0], **opts) if isinstance(opts, dict) else opts ) for k, opts in fs.items() } if None not in self.fss: self.fss[None] = filesystem("file") return ``` -------------------------------- ### Information Retrieval Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/sftp.html Provides methods to get information about files and directories. ```APIDOC ## Information Retrieval ### info Retrieves detailed information about a file or directory. * **path** (str) - Required - The path of the file or directory. Returns a dictionary containing file/directory information such as 'name', 'size', 'type', 'uid', 'gid', 'time', and 'mtime'. ``` -------------------------------- ### Create Directory Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/webhdfs.html Creates a directory at the specified path. ```python def mkdir(self, path, **kwargs): self._call("MKDIRS", method="put", path=path) ``` -------------------------------- ### Initialize LocalFileSystem with auto_mkdir Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/local.html Instantiates the LocalFileSystem. Set auto_mkdir to True to automatically create parent directories when opening a file. ```python fs = LocalFileSystem(auto_mkdir=True) ``` -------------------------------- ### Get Memory File Size Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/memory.html Returns the size of the memory file in bytes. ```python @property def size(self): return self.getbuffer().nbytes ``` -------------------------------- ### Get File Information Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/arrow.html Retrieves detailed information about a specific file or directory. ```python def info(self, path, **kwargs): path = self._strip_protocol(path) [info] = self.fs.get_file_info([path]) return self._make_entry(info) ``` -------------------------------- ### Setting up an aiohttp Client Session Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/http.html Asynchronously creates and sets an aiohttp.ClientSession if one does not already exist. Handles session closing when the file system is finalized. ```python async def set_session(self): if self._session is None: self._session = await self.get_client(loop=self.loop, **self.client_kwargs) if not self.asynchronous: weakref.finalize(self, self.close_session, self.loop, self._session) return self._session ``` -------------------------------- ### read_block Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html Reads a specified block of bytes from a file starting at a given offset. ```APIDOC ## read_block ### Description Read a block of bytes from a file starting at ``offset`` and reading ``length`` bytes. ### Parameters #### Path Parameters - **fn** (str) - Required - The file path. - **offset** (int) - Required - The starting byte offset to read from. - **length** (int) - Required - The number of bytes to read. - **delimiter** (str or None) - Optional - If provided, reading will stop at the first occurrence of this delimiter within the block. ### Method [Method details not explicitly provided, but typically a GET operation] ### Endpoint [Endpoint details not explicitly provided] ### Request Example ```json { "fn": "/path/to/large_file.bin", "offset": 1024, "length": 512, "delimiter": "\n" } ``` ### Response #### Success Response (200) - **block_data** (bytes) - The bytes read from the specified block. #### Response Example ```json { "block_data": "[binary data]" } ``` ``` -------------------------------- ### GistFileSystem Initialization Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/gist.html Initializes the GistFileSystem with a gist ID and optional parameters like filenames, revision (sha), authentication credentials, and request timeouts. It fetches the file list upon initialization. ```python import requests from ..spec import AbstractFileSystem from ..utils import infer_storage_options from .memory import MemoryFile class GistFileSystem(AbstractFileSystem): """ Interface to files in a single GitHub Gist. Provides read-only access to a gist's files. Gists do not contain subdirectories, so file listing is straightforward. Parameters ---------- gist_id: str The ID of the gist you want to access (the long hex value from the URL). filenames: list[str] (optional) If provided, only make a file system representing these files, and do not fetch the list of all files for this gist. sha: str (optional) If provided, fetch a particular revision of the gist. If omitted, the latest revision is used. username: str (optional) GitHub username for authentication. token: str (optional) GitHub personal access token (required if username is given), or. timeout: (float, float) or float, optional Connect and read timeouts for requests (default 60s each). kwargs: dict Stored on `self.request_kw` and passed to `requests.get` when fetching Gist metadata or reading ("opening") a file. """ protocol = "gist" gist_url = "https://api.github.com/gists/{gist_id}" gist_rev_url = "https://api.github.com/gists/{gist_id}/{sha}" def __init__( self, gist_id, filenames=None, sha=None, username=None, token=None, timeout=None, **kwargs, ): super().__init__() self.gist_id = gist_id self.filenames = filenames self.sha = sha # revision of the gist (optional) if username is not None and token is None: raise ValueError("User auth requires a token") self.username = username self.token = token self.request_kw = kwargs # Default timeouts to 60s connect/read if none provided self.timeout = timeout if timeout is not None else (60, 60) # We use a single-level "directory" cache, because a gist is essentially flat self.dircache[""] = self._fetch_file_list() ``` -------------------------------- ### download Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Alias for the `get` method, used to download files from the filesystem to a local path. ```APIDOC ## download ### Description Alias of `AbstractFileSystem.get`. ``` -------------------------------- ### Configure Local File System Defaults Source: https://filesystem-spec.readthedocs.io/en/latest/features.html Set default keyword arguments for the 'file' filesystem using a JSON configuration file. This example shows how to enable auto_mkdir for the LocalFileSystem. ```json {"file": {"auto_mkdir": true}} ``` -------------------------------- ### Get Delegation Token Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/webhdfs.html Retrieves a delegation token for authentication. Optionally specifies a renewer. ```python def get_delegation_token(self, renewer=None): """Retrieve token which can give the same authority to other uses Parameters ---------- renewer: str or None User who may use this token; if None, will be current user """ if renewer: out = self._call("GETDELEGATIONTOKEN", renewer=renewer) else: out = self._call("GETDELEGATIONTOKEN") t = out.json()["Token"] if t is None: raise ValueError("No token available for this user/security context") return t["urlString"] ``` -------------------------------- ### Get Filesystem Path from SMBFileOpener Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/smb.html Returns the definitive path of the file represented by the SMBFileOpener. ```python def __fspath__(self): return self.path ``` -------------------------------- ### Built-in Implementations Source: https://filesystem-spec.readthedocs.io/en/latest/api.html List of concrete filesystem implementations provided by the library, each with an `__init__` method. ```APIDOC ## Built-in Implementations Concrete filesystem implementations. ### `ArrowFSWrapper` * `__init__()` ### `HadoopFileSystem` * `__init__()` ### `CachingFileSystem` * `__init__()` ### `SimpleCacheFileSystem` * `__init__()` ### `WholeFileCacheFileSystem` * `__init__()` ### `DaskWorkerFileSystem` * `__init__()` ### `DataFileSystem` * `__init__()` ### `DatabricksFileSystem` * `__init__()` ### `DirFileSystem` * `__init__()` ### `FTPFileSystem` * `__init__()` ### `GistFileSystem` * `__init__()` ### `GitFileSystem` * `__init__()` ### `GithubFileSystem` * `__init__()` ### `HTTPFileSystem` * `__init__()` ### `JupyterFileSystem` * `__init__()` ### `LibArchiveFileSystem` * `__init__()` ### `LocalFileSystem` * `__init__()` ### `MemoryFileSystem` * `__init__()` ### `ReferenceFileSystem` * `__init__()` ### `LazyReferenceMapper` * `__init__()` ### `SFTPFileSystem` * `__init__()` ### `SMBFileSystem` * `__init__()` ### `TarFileSystem` * `__init__()` ### `WebHDFS` * `__init__()` ### `ZipFileSystem` * `__init__()` ``` -------------------------------- ### Get File Information Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/ftp.html Retrieves information about a specific file or directory on the FTP server. ```APIDOC ## info ### Description Retrieves information about a specific file or directory on the FTP server. ### Method `info(path, **kwargs)` ### Parameters * `path` (str) - The path to the file or directory. * `**kwargs` - Additional keyword arguments. ### Returns A dictionary containing information about the file or directory, including 'name', 'size', and 'type'. ``` -------------------------------- ### open Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Return a file-like object from the filesystem, usable in a context manager. ```APIDOC ## open(_path_ , _mode ='rb'_, _block_size =None_, _cache_options =None_, _compression =None_, _** kwargs_) ### Description Return a file-like object from the filesystem. The resultant instance must function correctly in a context `with` block. ### Parameters #### Path Parameters - **path** (str) - Required - Target file. - **mode** (str) - Optional - Defaults to 'rb'. Mode string like ‘rb’, ‘w’. Mode “x” (exclusive write) may be implemented by the backend. - **block_size** (int) - Optional - Some indication of buffering - this is a value in bytes. - **cache_options** (dict) - Optional - Extra arguments to pass through to the cache. - **compression** (string or None) - Optional - If given, open file using compression codec. Can either be a compression name (a key in `fsspec.compression.compr`) or “infer” to guess the compression from the filename suffix. - **encoding, errors, newline** - Optional - Passed on to TextIOWrapper for text mode. - **kwargs** - Optional - Additional keyword arguments. ``` -------------------------------- ### Implementing a Custom Async File System Method Source: https://filesystem-spec.readthedocs.io/en/latest/_sources/async.rst.txt Provides a template for implementing new asynchronous file system backends by deriving from AsyncFileSystem and defining async methods. It also shows how to expose these async methods via a synchronous wrapper. ```python class MyFileSystem(AsyncFileSystem): async def _my_method(self): ... my_method = sync_wrapper(_my_method) ``` -------------------------------- ### Get Sizes of Multiple Files Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html Efficiently retrieves the sizes in bytes for a list of file paths. ```python def sizes(self, paths): """Size in bytes of each file in a list of paths""" return [self.size(p) for p in paths] ``` -------------------------------- ### Initialize CachingFileSystem Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Wrap an existing filesystem with CachingFileSystem to enable local caching of remote files. This can improve performance by reducing repeated network access. ```python fs = CachingFileSystem(other_fs) ``` -------------------------------- ### Get File Size Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html Retrieves the size of a file in bytes. Returns None if the size is not available. ```python def size(self, path): """Size in bytes of file""" return self.info(path).get("size", None) ``` -------------------------------- ### Direct Async File Fetching and Session Management Source: https://filesystem-spec.readthedocs.io/en/latest/async.html Shows how to create an asynchronous filesystem instance within a coroutine and directly use its async methods like `_cat` for concurrent fetching. It also demonstrates explicit session creation and closing. ```python async def work_coroutine(): fs = fsspec.filesystem("http", asynchronous=True) session = await fs.set_session() # creates client out = await fs._cat([url1, url2, url3]) # fetches data concurrently await session.close() # explicit destructor asyncio.run(work_coroutine()) ``` -------------------------------- ### Get Home Directory Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/webhdfs.html Retrieves the user's home directory path using the WebHDFS API. ```python def home_directory(self): """Get user's home directory""" out = self._call("GETHOMEDIRECTORY") return out.json()["Path"] ``` -------------------------------- ### DirFileSystem Initialization Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/dirfs.html Initializes the DirFileSystem, wrapping an existing filesystem or creating one based on provided protocol and options. It ensures the underlying filesystem is compatible with the desired synchronization mode (async/sync). ```APIDOC ## DirFileSystem(path=None, fs=None, fo=None, target_protocol=None, target_options=None, **storage_options) ### Description Initializes the DirFileSystem, which wraps another filesystem and treats all paths as relative to a specified directory. ### Parameters #### Path Parameters - **path** (str) - Optional - Path to the directory to use as the root for this filesystem. - **fs** (AbstractFileSystem) - Optional - An instantiated filesystem to wrap. If not provided, one will be created using `target_protocol` and `target_options`. - **fo** (str) - Optional - An alternative to `path`. Do not provide both `path` and `fo`. - **target_protocol** (str) - Optional - The protocol of the filesystem to create if `fs` is not provided. - **target_options** (dict) - Optional - Options to pass to the underlying filesystem constructor if `fs` is not provided. - ** **storage_options** (dict) - Optional - Additional keyword arguments to be passed to the underlying filesystem constructor or the DirFileSystem itself. ``` -------------------------------- ### Get Filesystem Class by Protocol Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/registry.html Retrieves the filesystem class for a given protocol and instantiates it with storage options. ```python cls = get_filesystem_class(protocol) return cls(**storage_options) ``` -------------------------------- ### Get File System ID Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/arrow.html Generates a unique identifier for the file system based on its host and port. ```python @cached_property def fsid(self): return "hdfs_" + tokenize(self.fs.host, self.fs.port) ``` -------------------------------- ### SMBFileSystem Constructor Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/smb.html Initializes the SMBFileSystem, enabling connections to SMB shares. It supports various authentication methods and connection configurations, including optional parameters for timeouts, encryption, and share access control. ```APIDOC ## SMBFileSystem ### Description Initializes the SMBFileSystem, enabling connections to SMB shares. It supports various authentication methods and connection configurations, including optional parameters for timeouts, encryption, and share access control. ### Parameters * **host** (str) - The remote server name/ip to connect to. * **port** (int or None) - Port to connect with. Usually 445, sometimes 139. * **username** (str or None) - Username to connect with. Required if Kerberos auth is not being used. * **password** (str or None) - User's password on the server, if using username. * **timeout** (int) - Connection timeout in seconds. * **encrypt** (bool) - Whether to force encryption or not, once this has been set to True the session cannot be changed back to False. * **share_access** (str or None) - Specifies the default access applied to file open operations performed with this file system object. This affects whether other processes can concurrently open a handle to the same file. - None (the default): exclusively locks the file until closed. - 'r': Allow other handles to be opened with read access. - 'w': Allow other handles to be opened with write access. - 'd': Allow other handles to be opened with delete access. * **register_session_retries** (int) - Number of retries to register a session with the server. Retries are not performed for authentication errors, as they are considered as invalid credentials and not network issues. If set to negative value, no register attempts will be performed. * **register_session_retry_wait** (int) - Time in seconds to wait between each retry. Number must be non-negative. * **register_session_retry_factor** (int) - Base factor for the wait time between each retry. The wait time is calculated using exponential function. For factor=1 all wait times will be equal to `register_session_retry_wait`. For any number of retries, the last wait time will be equal to `register_session_retry_wait` and for retries>1 the first wait time will be equal to `register_session_retry_wait / factor`. Number must be equal to or greater than 1. Optimal factor is 10. * **auto_mkdir** (bool) - Whether, when opening a file, the directory containing it should be created (if it doesn't already exist). This is assumed by pyarrow and zarr-python code. ``` -------------------------------- ### Get File System Protocol Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/arrow.html Returns the protocol type name of the underlying pyarrow file system. ```python @property def protocol(self): return self.fs.type_name ``` -------------------------------- ### SFTPFileSystem Initialization Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/sftp.html Initializes the SFTPFileSystem to connect to an SFTP/SSH server. It takes the hostname and optional SSH connection parameters. ```APIDOC ## SFTPFileSystem ### Description Initializes the SFTPFileSystem to connect to an SFTP/SSH server using paramiko. ### Parameters * **host** (str) - Required - The hostname or IP address of the SFTP server. * **ssh_kwargs** (dict) - Optional - Additional parameters passed to paramiko's SSHClient.connect method. This can include 'port', 'username', 'password', etc. * **temppath** (str) - Optional - The path on the server to use for temporary files during transactions. Defaults to '/tmp'. ``` -------------------------------- ### Reference Filesystem Initialization with Single Remote Filesystem Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/reference.html Initializes the reference filesystem with a single remote filesystem. It also attempts to discover and add other necessary filesystems based on template or reference protocols. ```python if fs is not None: # single remote FS remote_protocol = ( fs.protocol[0] if isinstance(fs.protocol, tuple) else fs.protocol ) self.fss[remote_protocol] = fs if remote_protocol is None: # get single protocol from any templates for ref in self.templates.values(): if callable(ref): ref = ref() protocol, _ = fsspec.core.split_protocol(ref) if protocol and protocol not in self.fss: fs = filesystem(protocol, **(remote_options or {})) self.fss[protocol] = fs if remote_protocol is None: # get single protocol from references # TODO: warning here, since this can be very expensive? for ref in self.references.values(): if callable(ref): ref = ref() if isinstance(ref, list) and ref[0]: protocol, _ = fsspec.core.split_protocol(ref[0]) if protocol not in self.fss: fs = filesystem(protocol, **(remote_options or {})) self.fss[protocol] = fs # only use first remote URL break if remote_protocol and remote_protocol not in self.fss: fs = filesystem(remote_protocol, **(remote_options or {})) self.fss[remote_protocol] = fs self.fss[None] = fs or filesystem("file") # default one # Wrap any non-async filesystems to ensure async methods are available below for k, f in self.fss.items(): if not f.async_impl: self.fss[k] = AsyncFileSystemWrapper(f, asynchronous=self.asynchronous) elif self.asynchronous ^ f.asynchronous: raise ValueError( "Reference-FS's target filesystem must have same value " "of asynchronous" ) ``` -------------------------------- ### Get Filesystem Class by Protocol Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Fetches the specific filesystem implementation class registered for a given protocol name. ```python fsspec.get_filesystem_class(_protocol_) ``` -------------------------------- ### AbstractArchiveFileSystem info Method Source: https://filesystem-spec.readthedocs.io/en/latest/api.html Provides details of an entry at a given path. The default implementation calls ls and can be overridden for efficiency. It returns a dictionary with file information. ```python info(_path_ , _** kwargs_): Give details of entry at path Returns a single dictionary, with exactly the same information as `ls` would with `detail=True`. The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to ``ls() `. Some file systems might not be able to measure the file’s size, in which case, the returned dict will include `'size': None`. Returns: dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys. ``` -------------------------------- ### AbstractBufferedFile - full_name Property Source: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/spec.html Property to get the full path, including protocol, by stripping any leading protocol if present. ```python @property def full_name(self): return _unstrip_protocol(self.path, self.fs) ```