### Starting a Daemon Mount Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/error-daemon.md Use this example to fork the current process into a background daemon for a mount point. It includes setup for signaling readiness and running the event loop. ```rust use hf_mount::daemon::daemonize; use std::path::Path; async fn start_daemon_mount() -> hf_mount::error::Result<()> { let mount_point = Path::new("/tmp/mnt"); let source_label = "bucket myorg/mybucket"; // Fork and become a daemon let mut guard = daemonize(mount_point, source_label)?; // Child process continues here... // Do mount setup (Hub API, VirtualFs, FUSE/NFS, etc.) // ... // Signal parent that we're ready guard.notify_ready(); // Run event loop until shutdown // ... Ok(()) } ``` -------------------------------- ### Start hf-mount in Overlay Mode Source: https://github.com/huggingface/hf-mount/blob/main/README.md Use this command to start hf-mount in overlay mode. The producer starts a regular bucket mount, while the consumer mounts the same bucket with `--overlay` to read from the bucket and compile locally on cache misses. ```bash # Producer (writes compiled artifacts to the bucket — regular bucket mount) hf-mount start bucket myorg/torch-compile-cache "$TORCHINDUCTOR_CACHE_DIR" # Consumer (reads from the bucket, compiles locally on miss) hf-mount start --overlay bucket myorg/torch-compile-cache "$TORCHINDUCTOR_CACHE_DIR" ``` -------------------------------- ### Install hf-csi-driver with Helm Source: https://github.com/huggingface/hf-mount/blob/main/README.md Installs the hf-csi-driver using Helm. This is the recommended way to integrate hf-mount with Kubernetes. ```bash helm install hf-csi oci://ghcr.io/huggingface/charts/hf-csi-driver ``` -------------------------------- ### Start Bucket with Advanced Writes and Staging Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Mount a bucket with advanced write capabilities enabled. Configure debounce, batch window, and staging size for optimized performance. ```bash hf-mount start --advanced-writes \ --flush-debounce-ms 2000 \ --flush-max-batch-window-ms 30000 \ --max-staging-size 50000000 \ bucket myorg/mybucket /tmp/data ``` -------------------------------- ### Start hf-mount for a Hugging Face Bucket Source: https://github.com/huggingface/hf-mount/blob/main/README.md Mount a Hugging Face bucket as a local directory. The command picks up HF_TOKEN from the environment or can be passed explicitly. ```bash hf-mount start bucket myuser/my-bucket /tmp/data ``` -------------------------------- ### Manage Daemon Mounts Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Provides functions to start, list, and stop daemon mounts. Use `daemonize` to start a mount and `list_daemons` to see running mounts. `stop_daemon` can be used to terminate a specific mount. ```rust use hf_mount::daemon; use std::path::Path; // Start daemon let guard = daemon::daemonize( Path::new("/tmp/mnt"), "bucket myorg/mybucket" )?; guard.notify_ready(); // signal parent that mount is ready // ... continue with mount setup ... // List running mounts let daemons = daemon::list_daemons(); for d in daemons { println!("pid={} {} → {}", d.pid, d.source.unwrap_or_default(), d.mount_point); } // Stop a mount daemon::stop_daemon(Path::new("/tmp/mnt"))?; ``` -------------------------------- ### Mount NFS Filesystem Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Shows how to set up an NFS server using the NFS adapter. This example configures the adapter for read/write access and binds it to a TCP listener on the standard NFS port. ```rust use nfsserve::tcp::NFSTcpListener; use std::net::SocketAddr; async fn mount_nfs( vfs: Arc, ) -> hf_mount::error::Result<()> { let adapter = NFSAdapter::new(vfs, false); // read_only = false let listen_addr: SocketAddr = "0.0.0.0:2049".parse()?; let listener = NFSTcpListener::bind(listen_addr, adapter) .map_err(|e| hf_mount::error::Error::Io(e))?; listener.run().await; Ok(()) } ``` -------------------------------- ### Install hf-mount with Homebrew Source: https://github.com/huggingface/hf-mount/blob/main/README.md Installs the hf-mount package and its NFS backend on macOS and Linux using Homebrew. For FUSE backend on macOS, manual installation is required. ```bash brew install hf-mount ``` -------------------------------- ### Create Streaming Writer Example Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/xet-operations.md Demonstrates how to create a streaming writer for uploading data in chunks. Ensure the XetSessions object is mutable if you intend to write. ```rust let mut writer = xet_sessions.create_streaming_writer().await?; writer.write(b"data chunk 1").await?; writer.write(b"data chunk 2").await?; let file_info = writer.finish_boxed().await?; println!("Uploaded as hash: {}", file_info.xet_hash()); ``` -------------------------------- ### Start Bucket Mount with Token Rotation (Kubernetes CSI) Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Mount a bucket using a token file, suitable for Kubernetes CSI integration. Specify the path to the token file for secure access. ```bash hf-mount start \ --token-file /var/run/secrets/hf-token \ bucket myorg/mybucket /tmp/data ``` -------------------------------- ### Start Small Repo Mount (Read-Only) Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Mount a small Hugging Face Hub repository in read-only mode. Ensure the hub endpoint is correctly specified. ```bash hf-mount start repo openai-community/gpt2 /tmp/gpt2 \ --hub-endpoint https://huggingface.co ``` -------------------------------- ### Mount FUSE Filesystem Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Demonstrates how to mount a virtual file system using the FUSE adapter. This setup includes options for filesystem naming, subtype, and access control. ```rust use fuser::{MountOption, Session}; use std::path::Path; async fn mount_fuse( vfs: Arc, mount_point: &Path, ) -> hf_mount::error::Result<()> { let adapter = FuseAdapter::new( tokio::runtime::Handle::current(), vfs, Duration::from_secs(10), false, // read_only true, // advanced_writes false, // direct_io ); let session = Session::new( adapter, mount_point, &[ MountOption::FSName("hf-mount".to_string()), MountOption::Subtype("hf-bucket".to_string()), MountOption::AllowOther, ], )?; session.run().await; Ok(()) } ``` -------------------------------- ### Configure macOS LaunchAgent for hf-mount Source: https://github.com/huggingface/hf-mount/blob/main/README.md Creates a LaunchAgent plist file to automatically start hf-mount on login on macOS. Includes commands to bootstrap and bootout the agent. ```bash label=co.huggingface.hf-mount mkdir -p ~/Library/LaunchAgents cat > ~/Library/LaunchAgents/$label.plist < Label $label ProgramArguments $HOME/.local/bin/hf-mount-nfs repo openai/gpt-oss-20b /tmp/gpt-oss RunAtLoad KeepAlive StandardOutPath /tmp/hf-mount.log StandardErrorPath /tmp/hf-mount.log EOF launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/$label.plist ``` ```bash launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/$label.plist ``` -------------------------------- ### Daemon Management Methods Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/README.md Utilities for managing the background daemon process, including starting, listing, and stopping. ```rust daemonize(), list_daemons(), stop_daemon() ``` -------------------------------- ### Setup Bucket Mount Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Configures a mount point for an S3-compatible bucket. Use this when you need to access data stored in a bucket as if it were a local filesystem. Requires specifying bucket ID and mount point. ```rust use hf_mount::setup::{Source, MountOptions, CacheMode}; use std::path::PathBuf; async fn setup_bucket_mount() -> hf_mount::error::Result<()> { let source = Source::Bucket { bucket_id: "myorg/mybucket".to_string(), mount_point: PathBuf::from("/tmp/mnt"), }; let options = MountOptions { hf_token: Some("hf_abc123...".to_string()), token_file: None, hub_endpoint: "https://huggingface.co".to_string(), cache_dir: PathBuf::from("/tmp/hf-mount-cache"), uid: None, gid: None, read_only: false, advanced_writes: true, poll_interval_secs: 30, poll_listing_concurrency: 4, cache_size: 10_000_000_000, max_staging_size: 50_000_000, no_disk_cache: false, cache_mode: CacheMode::File, direct_io: false, metadata_ttl_ms: Some(10000), metadata_ttl_minimal: false, flush_debounce_ms: 2000, flush_max_batch_window_ms: 30000, flush_shutdown_timeout_ms: 45000, no_filter_os_files: false, inode_soft_limit: 50000, lru_sweep_interval_ms: 5000, fuse_owner_only: false, overlay: false, }; println!("Mount point: {}", source.mount_point().display()); println!("Label: {}", source.label()); Ok(()) } ``` -------------------------------- ### Setup Repository Mount with Subfolder Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Configures a mount point for a specific Hugging Face repository, including a subfolder. Use this to access a particular part of a repository. Note that repositories are mounted read-only. ```rust use hf_mount::setup::{Source, MountOptions}; use std::path::PathBuf; async fn setup_repo_mount() -> hf_mount::error::Result<()> { let source = Source::Repo { repo_id: "transformers/models/bert".to_string(), // will be split into ID + subfolder mount_point: PathBuf::from("/tmp/bert"), revision: "main".to_string(), }; let options = MountOptions { hf_token: None, // public repo token_file: None, hub_endpoint: "https://huggingface.co".to_string(), cache_dir: PathBuf::from("/tmp/hf-mount-cache"), uid: None, gid: None, read_only: true, // repos are read-only advanced_writes: false, poll_interval_secs: 30, poll_listing_concurrency: 4, cache_size: 10_000_000_000, max_staging_size: 0, no_disk_cache: false, cache_mode: CacheMode::Chunk, direct_io: false, // ... other defaults }; println!("Repo: {}", source.label()); Ok(()) } ``` -------------------------------- ### Get Filesystem Capabilities Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Returns the capabilities of the file system, indicating whether it is ReadOnly or ReadWrite. ```rust fn capabilities(&self) -> VFSCapabilities ``` -------------------------------- ### Mount and Interact with a Hugging Face Repository Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/hub-api.md Mounts a specified repository (model, dataset, etc.) from the Hugging Face Hub and lists its contents. This example demonstrates mounting a public repository without requiring a token. ```rust use hf_mount::hub_api::{HubApiClient, SourceKind, RepoType}; async fn mount_repo() -> hf_mount::error::Result> { let client = HubApiClient::from_source( "https://huggingface.co", None, // public repo, no token needed None, SourceKind::Repo { repo_id: "openai-community/gpt2".to_string(), repo_type: RepoType::Model, revision: "main".to_string(), }, String::new(), // mount entire repo "fuse", ).await?; // Enumerate files let entries = client.list_tree("").await?; println!("Found {} items", entries.len()); Ok(client) } ``` -------------------------------- ### Start hf-mount for a Hugging Face Repository (Read-Only) Source: https://github.com/huggingface/hf-mount/blob/main/README.md Mount a Hugging Face model or dataset repository as a local directory in read-only mode. The HF_TOKEN environment variable is used for authentication. ```bash hf-mount start repo openai/gpt-oss-20b /tmp/gpt-oss ``` -------------------------------- ### Start Memory-Constrained Container Mount Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Mount a bucket within a memory-constrained environment using FUSE. Configure cache size, inode limits, and poll concurrency. ```bash hf-mount start --fuse \ --cache-size 1000000000 \ --inode-soft-limit 10000 \ --poll-listing-concurrency 2 \ bucket myorg/mybucket /tmp/data ``` -------------------------------- ### Complete Virtual File System (VFS) Lifecycle Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Demonstrates the full lifecycle of setting up and using the Virtual File System (VFS). This includes initializing the Hub API client, setting up xet-core sessions, configuring VFS parameters, creating the VFS instance, performing a file attribute lookup, and finally shutting down the VFS. ```rust use hf_mount::hub_api::HubApiClient; use hf_mount::virtual_fs::{VirtualFs, VfsConfig}; use hf_mount::xet::XetSessions; use std::time::Duration; // Initialize Hub API client let hub_client = HubApiClient::from_source( "https://huggingface.co", Some("hf_abc123..."), None, SourceKind { repo_id, repo_type, revision }, String::new(), "nfs", ).await?; // Initialize xet-core sessions let xet_sessions = XetSessions::new( ctx, download_session, None, // no upload config for repos cas_client, None, // no chunk cache for NFS ); // Create VirtualFs let config = VfsConfig { read_only: true, advanced_writes: false, uid: 1000, gid: 1000, poll_interval_secs: 30, poll_listing_concurrency: 4, metadata_ttl: Duration::from_secs(10), serve_lookup_from_cache: true, filter_os_files: true, direct_io: false, flush_debounce: Duration::from_secs(2), flush_max_batch_window: Duration::from_secs(30), flush_shutdown_timeout: Duration::from_secs(45), inode_soft_limit: 0, lru_sweep_interval: Duration::from_secs(5), }; let vfs = VirtualFs::new( runtime.handle().clone(), config, hub_client, xet_sessions, None, // file_cache None, // staging_dir None, // overlay_backing ).await?; // Use vfs for filesystem ops... let attr = vfs.getattr(1).await?; // root // On shutdown vfs.shutdown().await; ``` -------------------------------- ### VirtualFs::new Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Initializes a new VirtualFs instance with provided configuration and clients. ```APIDOC ## VirtualFs::new ### Description Initialize a new VirtualFs instance. ### Parameters - `runtime`: tokio runtime handle for async operations - `config`: VfsConfig bundle - `hub_client`: Hub API client (implements HubOps trait) - `xet_sessions`: xet-core download/upload session provider - `file_cache`: optional whole-file cache (for `cache_mode=file`) - `staging_dir`: optional staging directory (required for advanced_writes) - `overlay_backing`: optional overlay layer (for `--overlay` mode) ### Returns `Result>` — initialized VirtualFs or error ### Example ```rust let vfs = VirtualFs::new( runtime.handle().clone(), config, hub_client, xet_sessions, None, // file_cache staging_dir, None, // overlay_backing ).await?; ``` ``` -------------------------------- ### StagingDir::file_size Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/xet-operations.md Gets the size of a staging file for a given inode. ```APIDOC ## StagingDir::file_size ### Description Returns the size of a staging file for a given inode. If the file is not present, it returns 0. ### Signature ```rust pub fn file_size(&self, ino: u64) -> u64 ``` ### Parameters #### Path Parameters - `ino` (u64) - Required - The inode number of the staging file. ``` -------------------------------- ### Get Path Prefix Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/hub-api.md Returns the subfolder prefix that this client instance is scoped to. ```rust pub fn path_prefix(&self) -> &str ``` -------------------------------- ### Mount Configuration from CLI Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Demonstrates how to parse command-line arguments to configure mount options, including source, cache size, read-only status, and advanced write capabilities. Requires `clap` for argument parsing. ```rust use hf_mount::setup::{Source, MountOptions}; use clap::Parser; #[derive(Parser)] struct Cli { #[command(subcommand)] source: Source, #[command(flatten)] options: MountOptions, } async fn mount_from_cli() -> hf_mount::error::Result<()> { let cli = Cli::parse(); let mount_point = cli.source.mount_point().to_path_buf(); let label = cli.source.label(); println!("Mounting {} at {}", label, mount_point.display()); println!(" Cache: {} bytes", cli.options.cache_size); println!(" Read-only: {}", cli.options.read_only); println!(" Advanced writes: {}", cli.options.advanced_writes); Ok(()) } ``` -------------------------------- ### Get Source Kind Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/hub-api.md Returns the source kind (bucket or repository) configured for this client. ```rust pub fn source(&self) -> &SourceKind ``` -------------------------------- ### Initialize New VirtualFs Instance Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Constructs a new VirtualFs instance. Ensure all required parameters like runtime, configuration, and clients are provided. ```rust pub async fn new( runtime: tokio::runtime::Handle, config: VfsConfig, hub_client: Arc, xet_sessions: Arc, file_cache: Option>, staging_dir: Option, overlay_backing: Option>, ) -> Result> ``` ```rust let vfs = VirtualFs::new( runtime.handle().clone(), config, hub_client, xet_sessions, None, // file_cache staging_dir, None, // overlay_backing ).await?; ``` -------------------------------- ### FUSE Get Attributes Operation Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Retrieves inode attributes such as size, mode, and modification time. ```rust fn getattr(&mut self, req: &Request<'_>, ino: INodeNo, reply: ReplyAttr) ``` -------------------------------- ### Get PID File Path from DaemonGuard Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/error-daemon.md Retrieves the path to the PID file managed by the DaemonGuard. ```rust pub fn pid_file(&self) -> &Path ``` -------------------------------- ### Initialize OverlayBacking Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Initializes the overlay backing from an open directory file descriptor. Ensure the provided file descriptor is at the mount point. ```rust pub fn new(fd: std::fs::File) -> Self ``` -------------------------------- ### NFSAdapter::open Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Opens a file for reading. If `create` is true, it will create the file if it does not exist. ```APIDOC ## NFSAdapter::open ### Description Opens a file for reading. If the `create` flag is set to true, the file will be created if it does not already exist. ### Method `open` ### Parameters - `id` (fileid3) - Required - The inode ID of the file to open. - `create` (bool) - Required - If true, create the file if it does not exist. ### Returns - `Result` - The inode ID of the opened file if successful, or an NFS error code. ``` -------------------------------- ### Get Root Directory Inode Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Retrieves the root inode number, which is always 1 for the NFS adapter. ```rust fn root_dir(&self) -> fileid3 ``` -------------------------------- ### Daemon Control Methods Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/README.md Methods for managing the background daemon process, including starting, listing, and stopping daemons. ```APIDOC ## Daemon Control Methods ### Description Methods for managing the background daemon process, including starting, listing, and stopping daemons. ### Methods - **daemonize**: Start the HF Mount daemon in the background. - **list_daemons**: List currently running HF Mount daemons. - **stop_daemon**: Stop a running HF Mount daemon. ``` -------------------------------- ### NFSAdapter::readdir Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Lists directory entries starting from a given cookie. This method uses cookie-based pagination, specific to NFS. ```APIDOC ## NFSAdapter::readdir ### Description Lists directory entries starting from a specified cookie. This method employs cookie-based pagination, which is an NFS-specific approach. ### Method `readdir` ### Parameters - `id` (fileid3) - Required - The inode ID of the directory to list entries from. - `cookie` (u64) - Required - The cookie to start listing entries from. Use 0 for the first entry. ### Returns - `Result` - A structure containing the directory entries and a new cookie for pagination, or an NFS error code. ``` -------------------------------- ### Test Mount Integration Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Write integration tests for hf-mount using Rust and Tokio. This snippet demonstrates how to set up a mock Hub client and initialize a VirtualFs instance for testing filesystem operations. ```rust #[cfg(test)] mod tests { use super::*; use hf_mount::test_mocks::*; // mock Hub client, xet-core #[tokio::test] async fn test_mount() { let mock_hub = MockHubOps::new(); let vfs = VirtualFs::new( runtime.handle().clone(), config, Arc::new(mock_hub), // inject mock xet_sessions, None, None, None, ).await.unwrap(); // Test filesystem operations... } } ``` -------------------------------- ### Get Readiness Notification Pipe File Descriptor Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/error-daemon.md Retrieves the write end of the pipe used for readiness notifications from the DaemonGuard. ```rust pub fn write_fd(&self) -> i32 ``` -------------------------------- ### Get Cached File Size Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/types-file-cache.md Returns the size of a cached file in bytes. Returns 0 if the file is not present in the cache. ```rust pub fn size(&self, hash: &str) -> u64 ``` -------------------------------- ### VirtualFs Core Methods Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/README.md Essential methods for creating, querying, and performing I/O operations on the virtual filesystem. ```rust VirtualFs::new() // Create filesystem vfs.lookup(), vfs.getattr() // Query metadata vfs.open(), vfs.read(), vfs.write(), vfs.release() // I/O vfs.create(), vfs.mkdir(), vfs.unlink() // Create/delete ``` -------------------------------- ### Get Human-Readable Label Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Generates a human-readable label for the source, useful for logging or display. This method is part of the Source enum. ```rust pub fn label(&self) -> String ``` -------------------------------- ### Build hf-mount from Source with All Features Source: https://github.com/huggingface/hf-mount/blob/main/README.md Builds the hf-mount project with both FUSE and NFS backends enabled. Ensure system dependencies for FUSE are met. ```bash cargo build --release --features fuse,nfs ``` -------------------------------- ### Get Mount Point Path Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Retrieves the mount point path for a given source. This method is part of the Source enum. ```rust pub fn mount_point(&self) -> &Path ``` -------------------------------- ### List Directory Entries Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Lists directory entries starting from a given cookie. This method uses cookie-based pagination, which is specific to NFS. ```rust async fn readdir( &self, id: fileid3, cookie: u64, ) -> Result ``` -------------------------------- ### List Directory Entries Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Reads directory entries starting from a specified offset. This is used by NFS/FUSE readdir implementations and returns a batch of entries. ```rust pub async fn readdir( &self, ino: u64, file_handle: u64, offset: u64, size: u32, ) -> Result ``` -------------------------------- ### FileRange Struct Definition Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/xet-operations.md Defines a byte range for partial file downloads. The 'end' field is exclusive, specifying the range as [start, end). ```rust pub struct FileRange { pub start: u64, pub end: u64, } ``` -------------------------------- ### NFSAdapter Constructor Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Initializes a new NFSAdapter instance. Use this to set up the adapter with a virtual file system and specify if it should operate in read-only mode. ```rust pub fn new(virtual_fs: Arc, read_only: bool) -> Self> ``` -------------------------------- ### Get Inode Attributes Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Retrieves the attributes (size, mode, mtime, etc.) for a given inode. Metadata is revalidated based on its Time-To-Live (TTL). ```rust pub async fn getattr(&self, ino: u64) -> Result ``` -------------------------------- ### MountSetup Structure Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md Defines the fully initialized state of a mount after processing arguments and creating components. It includes the source, hub client, virtual file system, and an optional file cache. ```rust pub struct MountSetup { pub source: SourceKind, pub hub_client: Arc, pub virtual_fs: Arc, pub file_cache: Option>, } ``` -------------------------------- ### Get Inode Attributes Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Fetches the attributes for a given inode ID. This includes information like file size, modification times, etc. ```rust async fn getattr( &self, id: fileid3, ) -> Result ``` -------------------------------- ### Configure VfsConfig from MountOptions Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/setup-configuration.md This snippet shows how to create a `VfsConfig` struct by mapping values from `MountOptions`. It handles default values and conditional logic for TTL durations. ```rust use std::time::Duration; use hf_mount::virtual_fs::VfsConfig; use hf_mount::setup::MountOptions; fn options_to_vfs_config(opts: &MountOptions) -> VfsConfig { let metadata_ttl = if opts.metadata_ttl_minimal { Duration::from_millis(0) } else { Duration::from_millis(opts.metadata_ttl_ms.unwrap_or(10000)) }; VfsConfig { read_only: opts.read_only, advanced_writes: opts.advanced_writes, uid: opts.uid.unwrap_or_else(|| unsafe { libc::getuid() }), gid: opts.gid.unwrap_or_else(|| unsafe { libc::getgid() }), poll_interval_secs: opts.poll_interval_secs, poll_listing_concurrency: opts.poll_listing_concurrency as usize, metadata_ttl, serve_lookup_from_cache: true, filter_os_files: !opts.no_filter_os_files, direct_io: opts.direct_io, flush_debounce: Duration::from_millis(opts.flush_debounce_ms), flush_max_batch_window: Duration::from_millis(opts.flush_max_batch_window_ms), flush_shutdown_timeout: Duration::from_millis(opts.flush_shutdown_timeout_ms), inode_soft_limit: opts.inode_soft_limit, lru_sweep_interval: Duration::from_millis(opts.lru_sweep_interval_ms), } } ``` -------------------------------- ### StagingCoordinator Lock Method Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Gets or creates the per-inode async lock. This lock is held across awaits for operations like download, unlink, and flush. ```rust pub(crate) fn lock(&self, ino: u64) -> Arc> ``` -------------------------------- ### Run Unit Tests Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/INDEX.md Execute unit tests for the library. Ensure the 'fuse' and 'nfs' features are enabled. ```bash cargo test --lib --features fuse,nfs ``` -------------------------------- ### Mount via FUSE Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Sets up a FUSE adapter and session for mounting the virtual file system. Requires a runtime handle, VFS instance, and mount point. Kernel cache invalidation can be registered. ```rust use hf_mount::fuse::FuseAdapter; use fuser::{MountOption, Session}; use std::time::Duration; let adapter = FuseAdapter::new( runtime.handle().clone(), vfs.clone(), Duration::from_secs(10), cli.options.read_only, cli.options.advanced_writes, cli.options.direct_io, ); let mut session = Session::new( adapter, &mount_point, &[ MountOption::FSName("hf-mount".to_string()), MountOption::AllowOther, ], )?; // Register kernel cache invalidation callback session.set_invalidator(...); session.run().await; ``` -------------------------------- ### Create an InodeEntry for a File Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/types-file-cache.md Demonstrates how to construct an InodeEntry representing a file with various metadata. This is useful for manually defining file system entries. ```rust use std::sync::Arc; use std::time::SystemTime; use hf_mount::virtual_fs::inode::{InodeEntry, InodeKind, ROOT_INODE}; fn create_file_entry() -> InodeEntry { InodeEntry { inode: 2, parent: ROOT_INODE, name: Arc::from("hello.txt"), full_path: Arc::from("hello.txt"), kind: InodeKind::File, size: 1024, mtime: SystemTime::now(), mode: 0o644, uid: 1000, gid: 1000, atime: SystemTime::now(), ctime: SystemTime::now(), nlink: 1, symlink_target: None, xet_hash: Some("abc123...".to_string()), staging_is_current: false, etag: None, dirty_generation: 0, children_loaded_at: None, children_from_remote: false, children: Vec::new(), child_index: Default::default(), pending_deletes: Vec::new(), last_revalidated: None, eviction: Default::default(), } } ``` -------------------------------- ### RepoType::api_prefix Method Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/hub-api.md Returns the path segment for API routes like `/api/{type}/{id}/…`. For example, `models` for `RepoType::Model`. ```rust pub fn api_prefix(&self) -> &'static str ``` -------------------------------- ### Build Virtual File System (VirtualFs) Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Constructs a `VirtualFs` instance with detailed configuration options. This includes read-only mode, advanced writes, user/group IDs, polling intervals, and cache settings. ```rust use hf_mount::virtual_fs::{VirtualFs, VfsConfig}; use std::time::Duration; let config = VfsConfig { read_only: cli.options.read_only, advanced_writes: cli.options.advanced_writes, uid: cli.options.uid.unwrap_or_else(|| unsafe { libc::getuid() }), gid: cli.options.gid.unwrap_or_else(|| unsafe { libc::getgid() }), poll_interval_secs: cli.options.poll_interval_secs, poll_listing_concurrency: cli.options.poll_listing_concurrency as usize, metadata_ttl: Duration::from_millis(cli.options.metadata_ttl_ms.unwrap_or(10000)), serve_lookup_from_cache: true, filter_os_files: !cli.options.no_filter_os_files, direct_io: cli.options.direct_io, flush_debounce: Duration::from_millis(cli.options.flush_debounce_ms), flush_max_batch_window: Duration::from_millis(cli.options.flush_max_batch_window_ms), flush_shutdown_timeout: Duration::from_millis(cli.options.flush_shutdown_timeout_ms), inode_soft_limit: cli.options.inode_soft_limit, lru_sweep_interval: Duration::from_millis(cli.options.lru_sweep_interval_ms), }; let vfs = VirtualFs::new( runtime.handle().clone(), config, hub_client, xet_sessions, file_cache, // Option> staging_dir, // Option overlay_backing, // Option> ).await?; ``` -------------------------------- ### Run Benchmarks Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/INDEX.md Execute benchmark tests, specifically the 'bench' test. Requires an HF_TOKEN and the 'fuse', 'nfs' features. Captures output. ```bash HF_TOKEN=... cargo test --release --features fuse,nfs --test bench -- --nocapture ``` -------------------------------- ### Read Byte Range from File Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Reads a specified number of bytes from a file starting at a given offset. Returns the data and an EOF flag, or an NFS error. ```rust async fn read( &self, id: fileid3, offset: u64, count: u32, ) -> Result ``` -------------------------------- ### mkdir Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Creates a new directory. Returns the attributes of the newly created directory. ```APIDOC ## mkdir ### Description Create a directory. ### Method `async fn` ### Parameters #### Path Parameters - `parent` (u64) - Required - parent directory inode - `name` (str) - Required - directory name - `mode` (u16) - Required - Unix mode bits (typically 0o40755) - `uid` (u32) - Required - UID - `gid` (u32) - Required - GID ### Returns `Result` — attributes of the new directory or error ``` -------------------------------- ### Cache a File Using FileCache Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/types-file-cache.md Shows how to use FileCache to either retrieve an existing cached file or download, cache, and open a new file. This is useful for efficient file access in applications. ```rust use hf_mount::file_cache::FileCache; use std::path::Path; async fn cache_file(hash: &str) -> hf_mount::error::Result<()> { let cache = FileCache::new(Path::new("/tmp/cache"), 1_000_000_000)?; // Try to open cached file if let Some(file) = cache.try_open(hash).await? { println!("Using cached file"); return Ok(()) } // Download and populate let file = cache.populate(hash, 5_000_000, |tmp_path| async { // Download to tmp_path std::fs::write(tmp_path, b"file data")?; Ok(()) }).await?; println!("Cached and opened: {:?}", file); Ok(()) } ``` -------------------------------- ### Get Default Modification Time Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/hub-api.md Returns the default modification time, typically derived from repository or bucket information. This is used when per-file modification times are not available. ```rust pub fn default_mtime(&self) -> SystemTime ``` -------------------------------- ### Open File for Reading or Creation Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/fuse-nfs-adapters.md Opens a file for reading. If the `create` flag is true, the file will be created if it does not exist. ```rust async fn open( &self, id: fileid3, create: bool, ) -> Result ``` -------------------------------- ### Mount a Public Repository Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Mounts a public Hugging Face repository. Ensure you have the necessary dependencies and set up the virtual file system. ```rust use hf_mount::setup::{Source, MountOptions}; use hf_mount::hub_api::HubApiClient; use hf_mount::virtual_fs::VirtualFs; async fn mount_public_repo( repo_id: &str, revision: &str, mount_point: &Path, ) -> Result> { let hub_client = HubApiClient::from_source( "https://huggingface.co", None, // public, no token None, SourceKind::Repo { repo_id: repo_id.to_string(), repo_type: RepoType::Model, revision: revision.to_string(), }, String::new(), // no subfolder "nfs", ).await?; // ... create vfs ... Ok(vfs) } ``` -------------------------------- ### Download File to Disk Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/xet-operations.md This method is used to download a complete file from Xet to a specified local path. It requires the content hash and expected file size. ```rust async fn download_to_file( &self, xet_hash: &str, file_size: u64, dest: &Path, ) -> Result<()> ``` -------------------------------- ### StagingDir File Size Method Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Returns the size of a staging file. Returns 0 if the file is not present. ```rust pub fn file_size(&self, ino: u64) -> u64 ``` -------------------------------- ### create Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Creates a new file within a directory. It returns the attributes of the newly created file and an open file handle. ```APIDOC ## create ### Description Create a new file in a directory. Returns attributes and open file handle. ### Method `async fn` ### Parameters #### Path Parameters - `parent` (u64) - Required - parent directory inode - `name` (str) - Required - new filename - `mode` (u16) - Required - Unix mode bits (0o100644 for regular file) - `uid` (u32) - Required - UID for the new file - `gid` (u32) - Required - GID for the new file - `truncate_size` (Option) - Optional - optional truncation size ### Returns `Result<(VirtualFsAttr, u64)>` — attributes and file handle, or error ``` -------------------------------- ### daemonize Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/error-daemon.md Forks the current process and turns it into a daemon. The parent process waits for a readiness signal from the child, while the child proceeds to set up the mount and manage its lifecycle. ```APIDOC ## daemonize ### Description Fork and become a daemon. Returns a guard held by the child. ### Method `daemonize(mount_point: &Path, source_label: &str) -> Result` ### Parameters - `mount_point` (`&Path`): Local directory to mount at. Used to derive log and PID filenames. - `source_label` (`&str`): Human-readable description of the source (e.g., "bucket myorg/mybucket"). ### Behavior 1. Creates directories for logs and PID files under `~/.hf-mount/`. 2. Forks the process: the parent reads from a readiness pipe, and the child proceeds. 3. Child process: redirects stdout/stderr to a log file, creates a PID file, and returns a `DaemonGuard`. 4. Parent process: waits for the `DaemonGuard` signal or a timeout, and exits with status 0 if successful. ### Returns - `Result` - A `DaemonGuard` instance if successful, or an error if the operation fails. The parent process exits immediately upon failure. ``` -------------------------------- ### Open Directory for Listing Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Opens a directory to enable listing its contents. Returns a file handle that can be used for subsequent readdir operations. ```rust pub async fn opendir(&self, ino: u64) -> Result ``` -------------------------------- ### Initialize Hub API Client Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Initializes the `HubApiClient` for interacting with the Hugging Face Hub. It requires endpoint, token, source kind, path prefix, and backend name. The source must exist. ```rust use hf_mount::hub_api::HubApiClient; use std::sync::Arc; let hub_client = HubApiClient::from_source( &cli.options.hub_endpoint, cli.options.hf_token.as_deref(), cli.options.token_file.clone(), source_kind, // SourceKind::Bucket or ::Repo path_prefix, "nfs", // backend name ).await?; // Validate the source exists hub_client.validate_path_prefix().await?; ``` -------------------------------- ### Build hf-mount from Source with FUSE Feature Source: https://github.com/huggingface/hf-mount/blob/main/README.md Builds the hf-mount project with the FUSE backend. Requires macFUSE on macOS or fuse3 on Linux. The pre-built binaries only need the runtime. ```bash cargo build --release --features fuse ``` -------------------------------- ### FileCache::new Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/types-file-cache.md Initializes a new file cache instance. It sets up the necessary directories and scans existing files to initialize the cache's state. ```APIDOC ## FileCache::new ### Description Initializes a file cache. Creates `files/` and `.tmp//` subdirectories, scans existing files, and seeds the LRU clock. ### Method `pub fn new(cache_dir: &Path, capacity: u64) -> Result>` ### Parameters - `cache_dir` (`&Path`): The root directory for the cache. - `capacity` (`u64`): The maximum total bytes to cache. ### Returns - `Result>`: An `Arc` to the initialized cache or an error if initialization fails. ``` -------------------------------- ### Parse Mount Configuration with Clap Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Parses command-line arguments for mount configuration using `clap`. Ensure `Source` and `MountOptions` are correctly defined in your application. ```rust use hf_mount::setup::{Source, MountOptions}; use clap::Parser; #[derive(Parser)] struct Cli { #[command(subcommand)] source: Source, #[command(flatten)] options: MountOptions, } let cli = Cli::parse(); ``` -------------------------------- ### Run Benchmarks for hf-mount Source: https://github.com/huggingface/hf-mount/blob/main/README.md Executes benchmarks for hf-mount, typically requiring a Hugging Face token and FUSE/NFS features enabled. ```bash # Benchmarks HF_TOKEN=... cargo test --release --features fuse,nfs --test bench -- --nocapture ``` -------------------------------- ### Populate Cache with a New File Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/types-file-cache.md Downloads a file, caches it, and returns an open file handle. It first checks if the file is already cached or being downloaded. If not, it creates a temporary file, downloads the content, atomically renames it to its final location, and triggers LRU eviction if necessary. ```rust pub async fn populate( &self, hash: &str, file_size: u64, download: impl Fn(&Path) -> F, ) -> Result where F: std::future::Future>, ``` -------------------------------- ### VirtualFs Configuration Structure Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Defines the configuration parameters for initializing the VirtualFs. Use this struct to set options like read-only mode, advanced write behavior, user/group IDs, polling intervals, and caching strategies. ```rust pub struct VfsConfig { pub read_only: bool, pub advanced_writes: bool, pub uid: u32, pub gid: u32, pub poll_interval_secs: u64, pub poll_listing_concurrency: usize, pub metadata_ttl: Duration, pub serve_lookup_from_cache: bool, pub filter_os_files: bool, pub direct_io: bool, pub flush_debounce: Duration, pub flush_max_batch_window: Duration, pub flush_shutdown_timeout: Duration, pub inode_soft_limit: usize, pub lru_sweep_interval: Duration, } ``` -------------------------------- ### VirtualFs::opendir Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/virtual-fs.md Opens a directory for listing entries and returns a file handle. ```APIDOC ## VirtualFs::opendir ### Description Open a directory for listing. Returns a file handle. ### Parameters - `ino`: directory inode number ### Returns `Result` — file handle or error ``` -------------------------------- ### OverlayBacking::create_dir Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Creates a new directory at the specified path within the overlay layer with the given Unix mode bits. ```APIDOC ## OverlayBacking::create_dir ### Description Create a directory with specified mode bits in the overlay layer. ### Method `create_dir(&self, full_path: &str, mode: u16) -> std::io::Result<()>` ### Parameters #### Path Parameters - **full_path** (string) - Required - The path for the new directory. - **mode** (u16) - Required - Unix mode bits (e.g., 0o755). ### Returns `std::io::Result<()>` - Ok on success, or an error. ``` -------------------------------- ### Create Directory in Overlay Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Creates a new directory at the specified path within the overlay layer, with given Unix mode bits. The path is relative to the overlay root. ```rust pub fn create_dir(&self, full_path: &str, mode: u16) -> std::io::Result<()> ``` -------------------------------- ### Mount Bucket (Read-Write) Source: https://github.com/huggingface/hf-mount/blob/main/README.md Mounts a Hugging Face Bucket for read-write access. Buckets are S3-like storage for mutable data. ```bash hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data ``` -------------------------------- ### Load Model from Mounted Repository using Transformers Source: https://github.com/huggingface/hf-mount/blob/main/README.md Use a locally mounted Hugging Face repository with the Transformers library. Files are read on demand, avoiding a full download. ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("/tmp/gpt-oss") # reads on demand, no download step ``` -------------------------------- ### OverlayBacking Constructor Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/overlay-modes.md Initializes the OverlayBacking, which provides a sandboxed writable layer for overlay mode. It requires an open directory file descriptor at the mount point. ```APIDOC ## OverlayBacking::new ### Description Initializes overlay backing from a directory file descriptor. ### Method `new(fd: std::fs::File)` ### Parameters #### Path Parameters - **fd** (std::fs::File) - Required - An open directory file descriptor (must be at the mount point). ### Returns `Self` - An initialized `OverlayBacking` instance. ``` -------------------------------- ### Mount via NFS Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/QUICKSTART.md Creates an NFS adapter and listener for exposing the virtual file system over NFS. It binds to a specified socket address and runs the listener. ```rust use hf_mount::nfs::NFSAdapter; use nfsserve::tcp::NFSTcpListener; use std::net::SocketAddr; let adapter = NFSAdapter::new(vfs.clone(), cli.options.read_only); let listen_addr: SocketAddr = "0.0.0.0:2049".parse()?; let listener = NFSTcpListener::bind(listen_addr, adapter)?; listener.run().await; ``` -------------------------------- ### DaemonGuard::notify_ready Source: https://github.com/huggingface/hf-mount/blob/main/_autodocs/error-daemon.md Signals the parent process that the mount is active and ready. This function should be called by the daemon child process to indicate successful initialization. ```APIDOC ## DaemonGuard::notify_ready ### Description Signal the parent process that the mount is active and ready. After calling, the parent can exit. ### Method `notify_ready` ### Parameters None ### Behavior 1. Writes a "R" byte to the pipe. 2. Closes the pipe. 3. Sets `notified = true` to prevent duplicate writes. ```