### Installation Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Instructions on how to install the `hfdownload` CLI tool globally. ```APIDOC ## Installation ```bash dotnet tool install -g ElBruno.HuggingFace.Downloader.Cli ``` After installation, the `hfdownload` command is available globally. ``` -------------------------------- ### Install the NuGet package Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Add the library to your .NET project using the dotnet CLI. ```bash dotnet add package ElBruno.HuggingFace.Downloader ``` -------------------------------- ### Install HuggingFace Downloader CLI Tool Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/README.md Install the global CLI tool for managing Hugging Face downloads from your terminal. ```bash dotnet tool install -g ElBruno.HuggingFace.Downloader.Cli ``` -------------------------------- ### Dependency Injection Setup for Hugging Face Downloader Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/README.md Register the HuggingFaceDownloader service with dependency injection in your .NET application. Configure options like timeout and then inject the downloader into your services. ```csharp builder.Services.AddHuggingFaceDownloader(options => { options.Timeout = TimeSpan.FromMinutes(60); }); // Then inject HuggingFaceDownloader in your services public class MyModelService(HuggingFaceDownloader downloader) { public async Task EnsureModelAsync() { await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "my-org/my-model", LocalDirectory = DefaultPathHelper.GetDefaultCacheDirectory("MyApp"), RequiredFiles = ["model.onnx", "tokenizer.json"] }); } } ``` -------------------------------- ### Download Model Files using CLI Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/README.md Example of using the `hfdownload` CLI command to download specific model files from a Hugging Face repository. ```bash # Download model files hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json # Check if files exist locally hfdownload check sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json # List cached models hfdownload list # Delete a cached model hfdownload delete sentence-transformers/all-MiniLM-L6-v2 # See all commands hfdownload --help ``` -------------------------------- ### Get Platform-Specific Cache Directory Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Retrieves the appropriate cache directory path for the current operating system, suitable for storing downloaded models. Also sanitizes model names for safe file system usage. ```csharp using ElBruno.HuggingFace; // Get platform-specific cache directory string cacheDir = DefaultPathHelper.GetDefaultCacheDirectory("MyMLApp"); Console.WriteLine($"Cache directory: {cacheDir}"); // Windows: C:\Users\{user}\AppData\Local\MyMLApp\models // Linux: /home/{user}/.local/share/MyMLApp/models // macOS: /Users/{user}/.local/share/MyMLApp/models // Sanitize model names for use as directory names string modelId = "sentence-transformers/all-MiniLM-L6-v2"; string safeName = DefaultPathHelper.SanitizeModelName(modelId); Console.WriteLine($"Sanitized: {safeName}"); // Output: sentence-transformers_all-MiniLM-L6-v2 // Combine for full model path string modelDir = Path.Combine(cacheDir, safeName); Console.WriteLine($"Full path: {modelDir}"); // Output: C:\Users\{user}\AppData\Local\MyMLApp\models\sentence-transformers_all-MiniLM-L6-v2 ``` -------------------------------- ### Build and Test Project from Source Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/README.md Clone the repository, navigate to the project directory, and use `dotnet build` and `dotnet test` commands to build and run tests for the HuggingFace.Downloader project. ```bash git clone https://github.com/elbruno/ElBruno.HuggingFace.Downloader.git cd ElBruno.HuggingFace.Downloader dotnet build ElBruno.HuggingFace.Downloader.slnx dotnet test ElBruno.HuggingFace.Downloader.slnx ``` -------------------------------- ### Download Files with HuggingFaceDownloader Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Initialize the downloader and perform a basic file download from a public repository. ```csharp using ElBruno.HuggingFace; // Create downloader with default options using var downloader = new HuggingFaceDownloader(); // Download files from a public repository await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = "./models/miniLM", RequiredFiles = ["onnx/model.onnx", "tokenizer.json"], OptionalFiles = ["tokenizer_config.json", "vocab.txt"] }); // Output: Files downloaded to ./models/miniLM/onnx/model.onnx and ./models/miniLM/tokenizer.json ``` -------------------------------- ### Show current configuration Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Displays all current configuration values and the location of the configuration file. ```bash hfdownload config show ``` -------------------------------- ### Provide Hugging Face token via command-line option Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Use the --token option with the download command to authenticate for private or gated repositories. ```bash hfdownload download my-org/private-model model.bin --token hf_your_token ``` -------------------------------- ### List Command Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md List all downloaded models currently stored in the local Hugging Face cache. ```APIDOC ## `list` — List downloaded models ### Description Displays a list of all models that have been downloaded and are currently stored in the local Hugging Face cache. ### Method `GET` (conceptual, as it's a CLI command) ### Endpoint `hfdownload list [options]` ### Parameters #### Query Parameters - **--cache-dir** (string) - Optional - Specifies the cache directory to scan. Defaults to the platform's default cache directory. - **--format** (string) - Optional - Specifies the output format. Allowed values are `table` (default) or `json`. ### Request Example ```bash # List models in table format (default) hfdownload list # List models in JSON format hfdownload list --format json ``` ### Response #### Success Response (200 OK) Returns a list of downloaded models, including their repository ID, number of files, total size, and last modified date. The format can be a table or JSON. #### Response Example (Table Format) ``` ┌──────────────────────────────┬───────┬──────────┬──────────────────┐ │ Model │ Files │ Size │ Last Modified │ ├──────────────────────────────┼───────┼──────────┼──────────────────┤ │ sentence-transformers_all-… │ 3 │ 85.2 MB │ 2025-01-15 10:30 │ │ microsoft_phi-4-mini-… │ 2 │ 2.1 GB │ 2025-01-14 09:15 │ └──────────────────────────────┴───────┴──────────┴──────────────────┘ ``` #### Response Example (JSON Format) ```json [ { "model": "sentence-transformers/all-MiniLM-L6-v2", "files": 3, "size": "85.2 MB", "lastModified": "2025-01-15 10:30" }, { "model": "microsoft/Phi-4-mini-instruct-onnx", "files": 2, "size": "2.1 GB", "lastModified": "2025-01-14 09:15" } ] ``` ``` -------------------------------- ### Authenticate for private repositories Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Provide authentication tokens via environment variables or explicit configuration options. ```csharp // Option A: Set the HF_TOKEN environment variable (recommended) // export HF_TOKEN=hf_your_token_here // Option B: Pass the token explicitly var downloader = new HuggingFaceDownloader(new HuggingFaceDownloaderOptions { AuthToken = "hf_your_token_here" }); ``` -------------------------------- ### Manage configuration with hfdownload config Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt View and update persistent settings such as the default cache directory, authentication tokens, and revision branches. ```bash hfdownload config show ``` ```bash hfdownload config set cache-dir /data/hf-models ``` ```bash hfdownload config set default-token hf_your_token_here ``` ```bash hfdownload config set default-revision v2.0 ``` ```bash hfdownload config set no-progress true ``` ```bash hfdownload config reset --force ``` -------------------------------- ### Build Hugging Face File URL Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Constructs a direct download URL for a file from the Hugging Face Hub. Includes input validation to prevent path traversal. ```csharp using ElBruno.HuggingFace; // Build download URL for a file string url = HuggingFaceUrlBuilder.GetFileUrl( repoId: "sentence-transformers/all-MiniLM-L6-v2", filePath: "onnx/model.onnx", revision: "main" ); Console.WriteLine(url); // Output: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx // Download from specific tag string tagUrl = HuggingFaceUrlBuilder.GetFileUrl( "microsoft/Phi-4-mini-instruct-onnx", "model.onnx", "v1.0" ); Console.WriteLine(tagUrl); // Output: https://huggingface.co/microsoft/Phi-4-mini-instruct-onnx/resolve/v1.0/model.onnx // Input validation - throws ArgumentException for invalid input try { HuggingFaceUrlBuilder.GetFileUrl("invalid-repo", "file.txt"); // Missing owner/ } catch (ArgumentException ex) { Console.WriteLine(ex.Message); // Output: Invalid repository ID format 'invalid-repo'. Expected format: 'owner/repo'. } ``` -------------------------------- ### Hugging Face Downloader with Authentication Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/README.md Configure the HuggingFaceDownloader with an authentication token to access private or gated repositories. The token can be set via environment variables or directly in the options. ```csharp var downloader = new HuggingFaceDownloader(new HuggingFaceDownloaderOptions { AuthToken = "hf_your_token_here" }); ``` -------------------------------- ### Configure HuggingFaceDownloaderOptions in C# Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Sets global downloader settings such as authentication, timeouts, and HTTP request behavior. ```csharp using ElBruno.HuggingFace; var options = new HuggingFaceDownloaderOptions { // Authentication - falls back to HF_TOKEN environment variable if null AuthToken = Environment.GetEnvironmentVariable("HF_TOKEN"), // HTTP timeout for large model downloads (default: 30 minutes) Timeout = TimeSpan.FromMinutes(120), // Issue HEAD requests before download for accurate progress (default: true) ResolveFileSizesBeforeDownload = true, // Custom User-Agent header UserAgent = "MyMLApp/2.0 (contact@example.com)" }; using var downloader = new HuggingFaceDownloader(options); // Download large model with extended timeout await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "microsoft/Phi-4-mini-instruct-onnx", LocalDirectory = "./models/phi4", RequiredFiles = ["model.onnx"] }); ``` -------------------------------- ### Use with Dependency Injection Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Register the downloader in the .NET service container and inject it into services. ```csharp // In Program.cs or Startup.cs builder.Services.AddHuggingFaceDownloader(options => { options.Timeout = TimeSpan.FromMinutes(60); options.ResolveFileSizesBeforeDownload = true; }); // In your service public class MyModelService(HuggingFaceDownloader downloader) { public async Task EnsureModelAsync() { await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "my-org/my-model", LocalDirectory = DefaultPathHelper.GetDefaultCacheDirectory("MyApp"), RequiredFiles = ["model.onnx", "tokenizer.json"] }); } } ``` -------------------------------- ### Download model files Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Download specific files from a Hugging Face repository to a local directory. ```csharp using ElBruno.HuggingFace; using var downloader = new HuggingFaceDownloader(); await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = "./models/miniLM", RequiredFiles = ["onnx/model.onnx", "tokenizer.json"], OptionalFiles = ["tokenizer_config.json", "vocab.txt"] }); ``` -------------------------------- ### Download Command Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Download files from a Hugging Face repository to a local directory or the platform's cache directory. ```APIDOC ## `download` — Download files from a Hugging Face repository ### Description Downloads specified files from a Hugging Face repository. Files can be downloaded to the default cache directory or a custom output directory. Supports authentication for private repositories. ### Method `POST` (conceptual, as it's a CLI command) ### Endpoint `hfdownload download [options]` ### Parameters #### Path Parameters - **repo-id** (string) - Required - Hugging Face repository ID (e.g., `microsoft/Phi-4-mini-instruct-onnx`) - **files** (string[]) - Required - One or more files to download, relative to the repo root #### Query Parameters - **-o, --output** (string) - Optional - Local directory for downloaded files. Defaults to the platform's cache directory. - **-r, --revision** (string) - Optional - Git revision (branch, tag, or commit SHA). Defaults to `main`. - **-t, --token** (string) - Optional - Hugging Face authentication token. Overrides the `HF_TOKEN` environment variable. - **--optional** (boolean) - Optional - Treat listed files as optional. If true, failures to download specific files will be skipped. Defaults to `false`. - **--no-progress** (boolean) - Optional - Suppress progress bar output. Defaults to `false`. - **-q, --quiet** (boolean) - Optional - Minimal output, showing only errors. Defaults to `false`. ### Request Example ```bash # Download specific files from a public repo hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json # Download to a specific directory hfdownload download microsoft/Phi-4-mini-instruct-onnx model.onnx -o ./my-models # Download from a private repo with token hfdownload download my-org/private-model weights.bin -t hf_your_token # Download optional files (skip on failure) hfdownload download my-org/model config.json --optional # Quiet mode for scripts hfdownload download my-org/model weights.bin -q ``` ### Response This command primarily operates via side effects (downloading files) and standard output/error streams. Success is indicated by the absence of errors and the presence of downloaded files. Error messages will be printed to stderr. #### Success Response (0 Exit Code) Files are downloaded to the specified or default location. #### Response Example (No specific JSON response, output is typically file paths or progress indicators. Errors are printed to stderr.) ``` -------------------------------- ### Configure HuggingFaceDownloader with Custom Options Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Use HuggingFaceDownloaderOptions to provide authentication tokens or modify request behavior for private repositories. ```csharp using ElBruno.HuggingFace; // Create downloader with authentication for private/gated repositories var options = new HuggingFaceDownloaderOptions { AuthToken = "hf_your_token_here", // Or set HF_TOKEN environment variable Timeout = TimeSpan.FromMinutes(60), ResolveFileSizesBeforeDownload = true, UserAgent = "MyApp/1.0" }; using var downloader = new HuggingFaceDownloader(options); await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "my-org/private-model", LocalDirectory = "./models/private", RequiredFiles = ["model.onnx", "config.json"], Revision = "v2.0" // Download from specific tag/branch }); ``` -------------------------------- ### Download Files from Hugging Face CLI Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Command-line interface to download files from a Hugging Face repository. Files are saved to the platform's default cache directory. ```bash # Basic download - files go to platform cache directory hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json ``` -------------------------------- ### Register HuggingFaceDownloader with DI Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Registers HuggingFaceDownloader as a singleton service. Configure timeout and file size resolution. AuthToken is read from HF_TOKEN env var if not explicitly set. ```csharp using ElBruno.HuggingFace; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; var builder = Host.CreateApplicationBuilder(args); // Register HuggingFaceDownloader with DI builder.Services.AddHuggingFaceDownloader(options => { options.Timeout = TimeSpan.FromMinutes(60); options.ResolveFileSizesBeforeDownload = true; // AuthToken automatically reads from HF_TOKEN env var if not set }); var host = builder.Build(); // Use in a service public class ModelService { private readonly HuggingFaceDownloader _downloader; public ModelService(HuggingFaceDownloader downloader) { _downloader = downloader; } public async Task EnsureModelReadyAsync(CancellationToken cancellationToken = default) { string cacheDir = DefaultPathHelper.GetDefaultCacheDirectory("MyApp"); if (!_downloader.AreFilesAvailable(["model.onnx"], cacheDir)) { await _downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = cacheDir, RequiredFiles = ["onnx/model.onnx", "tokenizer.json"] }, cancellationToken); } } } ``` -------------------------------- ### Download files from Hugging Face Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Download specific files from a repository with options for output directories, revisions, and authentication. ```bash hfdownload download [options] ``` ```bash # Download specific files from a public repo hfdownload download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json # Download to a specific directory hfdownload download microsoft/Phi-4-mini-instruct-onnx model.onnx -o ./my-models # Download from a private repo with token hfdownload download my-org/private-model weights.bin -t hf_your_token # Download optional files (skip on failure) hfdownload download my-org/model config.json --optional # Quiet mode for scripts hfdownload download my-org/model weights.bin -q ``` -------------------------------- ### Check Local File Availability with C# Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Verifies if specific files exist in a local directory before initiating a download. ```csharp using ElBruno.HuggingFace; using var downloader = new HuggingFaceDownloader(); string[] requiredFiles = ["onnx/model.onnx", "tokenizer.json"]; string localDir = "./models/miniLM"; bool ready = downloader.AreFilesAvailable(requiredFiles, localDir); if (ready) { Console.WriteLine("All model files are available locally"); } else { Console.WriteLine("Some files are missing, downloading..."); await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = localDir, RequiredFiles = requiredFiles }); } // Output: "All model files are available locally" or "Some files are missing, downloading..." ``` -------------------------------- ### Configure DownloadRequest in C# Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Defines parameters for a download operation, including repository ID, file lists, and progress reporting. ```csharp using ElBruno.HuggingFace; // Full DownloadRequest with all options var request = new DownloadRequest { // Required properties RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = "./models/miniLM", RequiredFiles = ["onnx/model.onnx", "tokenizer.json"], // Throws if download fails // Optional properties OptionalFiles = ["tokenizer_config.json", "vocab.txt"], // Silently skips on failure Revision = "main", // Branch, tag, or commit SHA (default: "main") UseAtomicWrites = true, // Write to .tmp file first (default: true) Progress = new Progress(p => Console.WriteLine(p.Message)) }; using var downloader = new HuggingFaceDownloader(); await downloader.DownloadFilesAsync(request); ``` -------------------------------- ### Show model details with hfdownload info Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Retrieve detailed information about a specific cached model, such as file paths and sizes. ```bash hfdownload info sentence-transformers/all-MiniLM-L6-v2 ``` ```bash hfdownload info sentence-transformers/all-MiniLM-L6-v2 --format json ``` -------------------------------- ### Download models with hfdownload Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Download specific files from Hugging Face repositories using various options like custom directories, authentication tokens, and quiet modes. ```bash hfdownload download microsoft/Phi-4-mini-instruct-onnx model.onnx -o ./my-models ``` ```bash hfdownload download my-org/private-model weights.bin --token hf_your_token_here ``` ```bash hfdownload download my-org/model model.onnx --revision v2.0 ``` ```bash hfdownload download my-org/model config.json vocab.txt --optional ``` ```bash hfdownload download my-org/model model.onnx -q ``` ```bash hfdownload download my-org/model model.onnx --no-progress ``` -------------------------------- ### Use platform-specific cache directories Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Utilize helper methods to resolve OS-appropriate cache paths and sanitize directory names. ```csharp // Returns OS-appropriate cache path: // Windows: %LOCALAPPDATA%/MyApp/models // Linux/macOS: ~/.local/share/MyApp/models string cacheDir = DefaultPathHelper.GetDefaultCacheDirectory("MyApp"); // Sanitize model names for use as directory names string safeName = DefaultPathHelper.SanitizeModelName("org/model-name"); // → "org_model-name" ``` -------------------------------- ### Project Directory Structure Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/ARCHITECTURE.md The file and folder organization of the ElBruno.HuggingFace.Downloader repository. ```text ElBruno.HuggingFace.Downloader/ ├── src/ │ └── ElBruno.HuggingFace.Downloader/ │ ├── HuggingFaceDownloader.cs # Core download engine │ ├── DownloadRequest.cs # Download configuration │ ├── DownloadProgress.cs # Progress reporting model │ ├── DownloadStage.cs # Progress stage enum │ ├── HuggingFaceDownloaderOptions.cs # Downloader configuration │ ├── HuggingFaceUrlBuilder.cs # URL construction │ ├── ByteFormatHelper.cs # Byte formatting utility │ ├── DefaultPathHelper.cs # Platform cache paths │ └── ServiceCollectionExtensions.cs # DI registration ├── tests/ │ └── ElBruno.HuggingFace.Downloader.Tests/ ├── docs/ │ ├── GETTING_STARTED.md │ ├── API_REFERENCE.md │ └── ARCHITECTURE.md ├── README.md └── LICENSE ``` -------------------------------- ### Show model information Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Display details for a specific cached model. ```bash hfdownload info [options] ``` ```bash hfdownload info sentence-transformers/all-MiniLM-L6-v2 ``` -------------------------------- ### Check file availability Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Verify if files exist locally or identify which files are missing from the local directory. ```csharp bool ready = downloader.AreFilesAvailable( ["onnx/model.onnx", "tokenizer.json"], "./models/miniLM"); if (!ready) { var missing = downloader.GetMissingFiles( ["onnx/model.onnx", "tokenizer.json"], "./models/miniLM"); Console.WriteLine($"Missing {missing.Count} files"); } ``` -------------------------------- ### Reset configuration to defaults Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Deletes the configuration file, reverting all settings to their default values. Use --force to bypass confirmation. ```bash hfdownload config reset [--force] ``` -------------------------------- ### List cached models Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Display a list of downloaded models in table or JSON format. ```bash hfdownload list [options] ``` ```bash hfdownload list # ┌──────────────────────────────┬───────┬──────────┬──────────────────┐ # │ Model │ Files │ Size │ Last Modified │ # ├──────────────────────────────┼───────┼──────────┼──────────────────┤ # │ sentence-transformers_all-… │ 3 │ 85.2 MB │ 2025-01-15 10:30 │ # │ microsoft_phi-4-mini-… │ 2 │ 2.1 GB │ 2025-01-14 09:15 │ # └──────────────────────────────┴───────┴──────────┴──────────────────┘ hfdownload list --format json ``` -------------------------------- ### Configuration Management Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Allows viewing, setting, and resetting configuration values for the Hugging Face Downloader tool. ```APIDOC ## CONFIGURATION ### Description Manages the configuration settings for the hfdownload tool. ### Subcommands #### `config show` Displays all current configuration values and the config file location. **Endpoint:** `hfdownload config show` #### `config set ` Sets a specific configuration key to a new value. **Endpoint:** `hfdownload config set ` **Available Keys:** - **cache-dir** (string) - Default cache directory. - **default-token** (string) - Default Hugging Face auth token. - **default-revision** (string) - Default Git revision. Defaults to `main`. - **no-progress** (boolean) - Suppress progress bars by default. Defaults to `false`. **Example:** ```bash hfdownload config set cache-dir /data/hf-models hfdownload config set default-revision v2.0 hfdownload config set no-progress true ``` #### `config reset [--force]` Resets all configuration to defaults by deleting the config file. Use `--force` to skip confirmation. ``` -------------------------------- ### HuggingFaceDownloaderOptions - Configuration Options Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Customizes the behavior of the Hugging Face Downloader, including authentication, timeouts, and network settings. ```APIDOC ## HuggingFaceDownloaderOptions Object ### Description Configures downloader behavior including authentication, timeouts, and progress tracking options. ### Method N/A (This is a data structure used for configuration) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **AuthToken** (string) - Optional - Authentication token for private repositories. Falls back to the HF_TOKEN environment variable if null. - **Timeout** (TimeSpan) - Optional - HTTP timeout for large model downloads. Defaults to 30 minutes. - **ResolveFileSizesBeforeDownload** (bool) - Optional - If true, issues HEAD requests before downloading to get accurate file sizes for progress reporting. Defaults to true. - **UserAgent** (string) - Optional - Custom User-Agent header to be sent with requests. ### Request Example ```csharp using ElBruno.HuggingFace; var options = new HuggingFaceDownloaderOptions { // Authentication - falls back to HF_TOKEN environment variable if null AuthToken = Environment.GetEnvironmentVariable("HF_TOKEN"), // HTTP timeout for large model downloads (default: 30 minutes) Timeout = TimeSpan.FromMinutes(120), // Issue HEAD requests before download for accurate progress (default: true) ResolveFileSizesBeforeDownload = true, // Custom User-Agent header UserAgent = "MyMLApp/2.0 (contact@example.com)" }; using var downloader = new HuggingFaceDownloader(options); // Download large model with extended timeout await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "microsoft/Phi-4-mini-instruct-onnx", LocalDirectory = "./models/phi4", RequiredFiles = ["model.onnx"] }); ``` ### Response This object is used for configuration, not for responses. ``` -------------------------------- ### Cache Directory Configuration Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Information about the default cache directory locations and how to override them. ```APIDOC ## CACHE DIRECTORY ### Description Details the default cache directory locations and how to specify a custom one. ### Default Paths | Platform | |----------|----------| | Windows | `%LOCALAPPDATA%\hfdownload\models` | | Linux | `~/.local/share/hfdownload/models` | | macOS | `~/Library/Application Support/hfdownload/models` | ### Customization Override the default cache directory using the `--cache-dir` option on any command, or set a persistent default via `hfdownload config set cache-dir `. ``` -------------------------------- ### Track download progress Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Monitor the download process using the Progress class. ```csharp var progress = new Progress(p => { switch (p.Stage) { case DownloadStage.Checking: Console.WriteLine($"🔍 {p.Message}"); break; case DownloadStage.Downloading: Console.Write($"\r⬇️ [{p.CurrentFileIndex}/{p.TotalFileCount}] {p.CurrentFile} — {p.PercentComplete:F0}%"); break; case DownloadStage.Validating: Console.WriteLine($"\n✅ {p.Message}"); break; case DownloadStage.Complete: Console.WriteLine($"🎉 {p.Message}"); break; } }); await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = "./models/miniLM", RequiredFiles = ["onnx/model.onnx", "tokenizer.json"], Progress = progress }); ``` -------------------------------- ### List cached models with hfdownload list Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Display a list of all cached models, including file counts and sizes, with support for JSON output and custom cache directories. ```bash hfdownload list ``` ```bash hfdownload list --format json ``` ```bash hfdownload list --cache-dir /data/hf-models ``` -------------------------------- ### Set Hugging Face token via environment variable Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Recommended method for authentication. Set the HF_TOKEN environment variable with your Hugging Face access token. ```bash export HF_TOKEN=hf_your_token_here ``` -------------------------------- ### Verify cached files with hfdownload check Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Verify the presence of files in the local cache. Returns exit code 0 if all files are present, or 1 if any are missing. ```bash hfdownload check sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json ``` ```bash hfdownload check sentence-transformers/all-MiniLM-L6-v2 model.onnx -o ./custom-cache ``` ```bash if hfdownload check my-org/model model.onnx; then echo "Model ready" else echo "Model needs download" hfdownload download my-org/model model.onnx fi ``` -------------------------------- ### Download from a specific branch or tag Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Specify a branch, tag, or commit SHA using the Revision property. ```csharp await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "my-org/my-model", LocalDirectory = "./models", RequiredFiles = ["model.onnx"], Revision = "v2.0" // branch, tag, or commit SHA }); ``` -------------------------------- ### Set default Hugging Face token in configuration Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Store your Hugging Face access token persistently in the tool's configuration file. ```bash hfdownload config set default-token hf_your_token_here ``` -------------------------------- ### Check Command Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Check if specified files exist in the local cache for a given Hugging Face repository. ```APIDOC ## `check` — Check if files exist in the local cache ### Description Verifies the presence of specified files within the local Hugging Face cache for a given repository. Outputs indicators for each file (✅ for present, ❌ for missing) and a summary. ### Method `GET` (conceptual, as it's a CLI command) ### Endpoint `hfdownload check [options]` ### Parameters #### Path Parameters - **repo-id** (string) - Required - Hugging Face repository ID. - **files** (string[]) - Required - One or more files to check for existence in the cache. #### Query Parameters - **-o, --output** (string) - Optional - Local directory to check. Defaults to the platform's default cache directory. ### Request Example ```bash hfdownload check sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json ``` ### Response #### Success Response (0 Exit Code) Outputs status indicators for each file and a summary of files present. An exit code of `0` signifies all requested files were found. #### Response Example ``` # ✅ onnx/model.onnx # ❌ tokenizer.json # 1 of 2 files present ``` #### Error Response (1 Exit Code) An exit code of `1` indicates that one or more files were not found in the cache. ``` -------------------------------- ### Set configuration values Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Modify configuration settings by providing a key and its new value. This can be used to change the cache directory, default token, default revision, or progress bar behavior. ```bash hfdownload config set ``` ```bash hfdownload config set cache-dir /data/hf-models ``` ```bash hfdownload config set default-revision v2.0 ``` ```bash hfdownload config set no-progress true ``` -------------------------------- ### Atomic Write File Lifecycle Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/ARCHITECTURE.md Visual representation of the atomic write process where files are downloaded to a temporary location before being finalized. ```text model.onnx.tmp → (download complete) → model.onnx ``` -------------------------------- ### Authentication Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Details on how to authenticate with Hugging Face for accessing private or gated repositories. ```APIDOC ## AUTHENTICATION ### Description Supports Hugging Face authentication for private and gated repositories. ### Methods 1. **Environment Variable (Recommended):** Set the `HF_TOKEN` environment variable. ```bash export HF_TOKEN=hf_your_token_here ``` 2. **Command-line Option:** Use the `--token` option with the `download` command. ```bash hfdownload download my-org/private-model model.bin --token hf_your_token ``` 3. **Persistent Configuration:** Store the token in the configuration file using `config set`. ```bash hfdownload config set default-token hf_your_token_here ``` ``` -------------------------------- ### Info Command Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Show detailed information about a specific cached model. ```APIDOC ## `info` — Show details of a cached model ### Description Retrieves and displays detailed information about a specific model that has been downloaded and is present in the local Hugging Face cache. ### Method `GET` (conceptual, as it's a CLI command) ### Endpoint `hfdownload info [options]` ### Parameters #### Path Parameters - **repo-id** (string) - Required - The repository ID of the cached model for which to display details. #### Query Parameters - **--cache-dir** (string) - Optional - Specifies the cache directory to scan. Defaults to the platform's default cache directory. - **--format** (string) - Optional - Specifies the output format. Allowed values are `table` (default) or `json`. ### Request Example ```bash # Show info for a model in table format (default) hfdownload info sentence-transformers/all-MiniLM-L6-v2 # Show info for a model in JSON format hfdownload info microsoft/Phi-4-mini-instruct-onnx --format json ``` ### Response #### Success Response (200 OK) Returns detailed information about the specified cached model, including files, size, and last modified date. The output format can be a table or JSON. #### Response Example (Table Format) ``` ┌──────────────────────────────┬───────┬──────────┬──────────────────┐ │ Model │ Files │ Size │ Last Modified │ ├──────────────────────────────┼───────┼──────────┼──────────────────┤ │ sentence-transformers_all-… │ 3 │ 85.2 MB │ 2025-01-15 10:30 │ └──────────────────────────────┴───────┴──────────┴──────────────────┘ ``` #### Response Example (JSON Format) ```json { "model": "sentence-transformers/all-MiniLM-L6-v2", "files": 3, "size": "85.2 MB", "lastModified": "2025-01-15 10:30" } ``` ``` -------------------------------- ### Format Byte Counts to Human-Readable Strings Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Converts byte counts into human-readable strings with appropriate units (B, KB, MB, GB). Useful for displaying file sizes or download progress. ```csharp using ElBruno.HuggingFace; // Format various byte sizes Console.WriteLine(ByteFormatHelper.FormatBytes(512)); // "512 B" Console.WriteLine(ByteFormatHelper.FormatBytes(1536)); // "1.5 KB" Console.WriteLine(ByteFormatHelper.FormatBytes(1_500_000)); // "1.4 MB" Console.WriteLine(ByteFormatHelper.FormatBytes(2_500_000_000)); // "2.33 GB" // Use in progress reporting var progress = new Progress(p => { if (p.Stage == DownloadStage.Downloading) { string downloaded = ByteFormatHelper.FormatBytes(p.BytesDownloaded); string total = ByteFormatHelper.FormatBytes(p.TotalBytes); Console.WriteLine($"Downloaded {downloaded} of {total} ({p.PercentComplete:F1}%)"); } }); // Output: Downloaded 45.2 MB of 89.3 MB (50.6%) ``` -------------------------------- ### List Missing Files with C# Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Identifies which files from a provided list are absent from the local directory. ```csharp using ElBruno.HuggingFace; using var downloader = new HuggingFaceDownloader(); string[] allFiles = ["onnx/model.onnx", "tokenizer.json", "vocab.txt", "config.json"]; string localDir = "./models/miniLM"; IReadOnlyList missing = downloader.GetMissingFiles(allFiles, localDir); Console.WriteLine($"Missing {missing.Count} of {allFiles.Length} files:"); foreach (var file in missing) { Console.WriteLine($" - {file}"); } // Output: // Missing 2 of 4 files: // - vocab.txt // - config.json ``` -------------------------------- ### Check local cache status Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Verify if specific files exist in the local cache, returning exit code 0 if all are present. ```bash hfdownload check [options] ``` ```bash hfdownload check sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json # ✅ onnx/model.onnx # ❌ tokenizer.json # 1 of 2 files present ``` -------------------------------- ### Workflow Execution Flow Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/publishing.md Visual representation of the automated publishing process using OIDC tokens. ```text GitHub Release created (e.g. v1.0.0) → GitHub Actions triggers publish.yml → Builds + tests the project → Packs ElBruno.HuggingFace.Downloader.nupkg → Requests an OIDC token from GitHub → Exchanges the token with NuGet.org for a temporary API key (valid 1 hour) → Pushes the package to NuGet.org → Temp key expires automatically ``` -------------------------------- ### HuggingFaceUrlBuilder Helper Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/API_REFERENCE.md Static helper for generating Hugging Face file URLs. ```APIDOC ## HuggingFaceUrlBuilder ### Methods - **GetFileUrl(string repoId, string filePath, string revision = "main")** - Returns the HF download URL for a file. ``` -------------------------------- ### DownloadRequest Configuration Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/API_REFERENCE.md Defines the parameters for a file download operation. ```APIDOC ## DownloadRequest ### Properties - **RepoId** (string) - Required - HF repository ID (e.g., "sentence-transformers/all-MiniLM-L6-v2") - **LocalDirectory** (string) - Required - Local directory for downloaded files - **RequiredFiles** (IReadOnlyList) - Required - Files that must be downloaded (failure throws) - **OptionalFiles** (IReadOnlyList?) - Optional - Files downloaded on best-effort basis - **Revision** (string) - Optional - Git branch, tag, or commit SHA (Default: "main") - **Progress** (IProgress?) - Optional - Progress reporter - **UseAtomicWrites** (bool) - Optional - Write to temp file first, then rename (Default: true) ``` -------------------------------- ### HuggingFaceDownloader Class Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/API_REFERENCE.md The primary class for managing file downloads from Hugging Face repositories. ```APIDOC ## HuggingFaceDownloader ### Description The main entry point for downloading files from Hugging Face Hub repositories. Implements IDisposable. ### Methods - **DownloadFilesAsync(DownloadRequest, CancellationToken)** (Task) - Downloads files described by the request. Skips existing files. - **GetMissingFiles(IEnumerable, string)** (IReadOnlyList) - Returns files that don't exist in the local directory. - **AreFilesAvailable(IEnumerable, string)** (bool) - Returns true if all files exist locally. - **Dispose()** (void) - Disposes the HttpClient if owned by this instance. ``` -------------------------------- ### Configuration File Location Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/CLI_REFERENCE.md Information about the location of the configuration file on different operating systems. ```APIDOC ## CONFIGURATION FILE LOCATION ### Description Specifies the path to the configuration file on various operating systems. | Platform | |----------|----------| | Windows | `%APPDATA%\hfdownload\config.json` | | Linux | `~/.config/hfdownload/config.json` | | macOS | `~/.config/hfdownload/config.json` | ``` -------------------------------- ### Disable atomic writes Source: https://github.com/elbruno/elbruno.huggingface.downloader/blob/main/docs/GETTING_STARTED.md Configure the download request to write files directly, which is faster but lacks protection against partial downloads. ```csharp await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "my-org/my-model", LocalDirectory = "./models", RequiredFiles = ["model.onnx"], UseAtomicWrites = false // Write directly (faster, but no protection against partial downloads) }); ``` -------------------------------- ### DownloadRequest - Request Configuration Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Defines the configuration for a file download operation, including repository details, local paths, and file requirements. ```APIDOC ## DownloadRequest Object ### Description Describes a complete download request including repository ID, local directory, required/optional files, revision, and progress reporting options. ### Method POST (conceptual, as it's used in a method call) ### Endpoint N/A (This is a data structure, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **RepoId** (string) - Required - The identifier of the repository on Hugging Face. - **LocalDirectory** (string) - Required - The local path where files should be saved. - **RequiredFiles** (string[]) - Required - An array of filenames that must be downloaded successfully. The download will fail if any of these are not found or cannot be downloaded. - **OptionalFiles** (string[]) - Optional - An array of filenames that are desired but not critical. The download will proceed even if these files are not found or cannot be downloaded. - **Revision** (string) - Optional - The branch, tag, or commit SHA to download from. Defaults to "main". - **UseAtomicWrites** (bool) - Optional - If true, files are written to a temporary file first and then renamed upon completion. Defaults to true. - **Progress** (IProgress) - Optional - An object to report download progress. ### Request Example ```csharp using ElBruno.HuggingFace; // Full DownloadRequest with all options var request = new DownloadRequest { // Required properties RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = "./models/miniLM", RequiredFiles = ["onnx/model.onnx", "tokenizer.json"], // Throws if download fails // Optional properties OptionalFiles = ["tokenizer_config.json", "vocab.txt"], // Silently skips on failure Revision = "main", // Branch, tag, or commit SHA (default: "main") UseAtomicWrites = true, // Write to .tmp file first (default: true) Progress = new Progress(p => Console.WriteLine(p.Message)) }; using var downloader = new HuggingFaceDownloader(); await downloader.DownloadFilesAsync(request); ``` ### Response This object is used for requests, not for responses. ``` -------------------------------- ### AreFilesAvailable - Check Local File Availability Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Checks if all specified files exist in the local directory. This is useful for determining if a download is necessary before initiating one. ```APIDOC ## AreFilesAvailable ### Description Returns true if all specified files exist in the local directory. Useful for checking if a download is needed before attempting it. ### Method GET (conceptual, as it's a method call on an object) ### Endpoint N/A (This is a library method, not a REST endpoint) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```csharp using ElBruno.HuggingFace; using var downloader = new HuggingFaceDownloader(); string[] requiredFiles = ["onnx/model.onnx", "tokenizer.json"]; string localDir = "./models/miniLM"; bool ready = downloader.AreFilesAvailable(requiredFiles, localDir); if (ready) { Console.WriteLine("All model files are available locally"); } else { Console.WriteLine("Some files are missing, downloading..."); await downloader.DownloadFilesAsync(new DownloadRequest { RepoId = "sentence-transformers/all-MiniLM-L6-v2", LocalDirectory = localDir, RequiredFiles = requiredFiles }); } ``` ### Response #### Success Response (200) - **ready** (bool) - True if all files are available, false otherwise. #### Response Example ```json { "ready": true } ``` ``` -------------------------------- ### Clear cache with hfdownload purge Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Remove all cached models from the cache directory. Use --force to skip confirmation. ```bash hfdownload purge ``` ```bash hfdownload purge --force ``` ```bash hfdownload purge --cache-dir /data/hf-models --force ``` -------------------------------- ### Delete cached models with hfdownload delete Source: https://context7.com/elbruno/elbruno.huggingface.downloader/llms.txt Remove cached models from the system. Use the --force flag to bypass confirmation prompts. ```bash hfdownload delete sentence-transformers/all-MiniLM-L6-v2 ``` ```bash hfdownload delete sentence-transformers/all-MiniLM-L6-v2 --force ``` ```bash hfdownload delete my-org/model --cache-dir /data/hf-models --force ```