Hugging Face Hub Python Library

https://huggingface.co/docs/huggingface_hub/package_reference/file_download
Admin
Hugging Face Hub Python Library is a client library for interacting with the Hugging Face Hub,...
Tokens:4,847
Snippets:26
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Cache File Layout Example

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Illustrates the directory structure of the Hugging Face Hub cache, showing how files are organized by repository, revision, and blob identifiers.

```bash
. 
└── [ 160]  models--julien-c--EsperBERTo-small
    ├── [ 160]  blobs
    │   ├── [321M]  403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
    │   ├── [ 398]  7cb18dc9bafbfcf74629a4b760af1b160957a83e
```

--------------------------------

### Get HF File Metadata

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Fetches metadata for a file versioned on the Hugging Face Hub given its URL. This includes information like commit hash, ETag, location, and size.

```APIDOC
## Get metadata about a file

### get_hf_file_metadata

Fetch metadata of a file versioned on the Hub for a given url.

#### Parameters

* **url** (`str`) — File url, for example returned by hf_hub_url().
* **token** (`str` or `bool`, _optional_) — A token to be used for the download. If `True`, the token is read from the HuggingFace config folder. If `False` or `None`, no token is provided. If a string, it’s used as the authentication token.
* **timeout** (`float`, _optional_ , defaults to 10) — How many seconds to wait for the server to send metadata before giving up.
* **library_name** (`str`, _optional_) — The name of the library to which the object corresponds.
* **library_version** (`str`, _optional_) — The version of the library.
* **user_agent** (`dict`, `str`, _optional_) — The user-agent info in the form of a dictionary or a string.
* **headers** (`dict`, _optional_) — Additional headers to be sent with the request.
* **endpoint** (`str`, _optional_) — Endpoint of the Hub. Defaults to https://huggingface.co.
* **retry_on_errors** (`bool`, _optional_ , defaults to `False`) — Whether to retry on errors (429, 5xx, timeout, network errors). If False, no retry for fast fallback to local cache.

### HfFileMetadata

Data structure containing information about a file versioned on the Hub. Returned by get_hf_file_metadata() based on a URL.

#### Parameters

* **commit_hash** (`str`, _optional_) — The commit_hash related to the file.
* **etag** (`str`, _optional_) — Etag of the file on the server.
* **location** (`str`) — Location where to download the file. Can be a Hub url or not (CDN).
* **size** (`size`) — Size of the file. In case of an LFS file, contains the size of the actual LFS file, not the pointer.
* **xet_file_data** (`XetFileData`, _optional_) — Xet information for the file. This is only set if the file is stored using Xet storage.
```

--------------------------------

### File Download Parameters

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

This snippet outlines the extensive parameters available for downloading files from the Hugging Face Hub. It covers repository identification, file specifics, revision control, caching, authentication, and progress bar customization.

```python
def hf_hub_download(
    repo_id: str,
    filename: str,
    subfolder: str | None = None,
    repo_type: str | None = None,
    revision: str | None = None,
    library_name: str | None = None,
    library_version: str | None = None,
    cache_dir: str | pathlib.Path | None = None,
    local_dir: str | pathlib.Path | None = None,
    user_agent: dict | str | None = None,
    force_download: bool = False,
    etag_timeout: float = 10,
    token: bool | str | None = None,
    local_files_only: bool = False,
    headers: dict[str, str] | None = None,
    endpoint: str | None = None,
    tqdm_class: type[tqdm.asyncio.tqdm_asyncio] | None = None,
    dry_run: bool = False,
) -> str or DryRunFileInfo:
```

--------------------------------

### Construct Hugging Face Hub File URL

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Use this function to generate a direct URL to a file in a Hugging Face repository. It handles routing to Cloudfront for large files to ensure faster downloads.

```python
from huggingface_hub import hf_hub_url

hf_hub_url(
    repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin"
)
```

--------------------------------

### huggingface_hub.file_download

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Downloads a file from the Hugging Face Hub. It can handle various repository types and revisions, with options for caching, force downloading, and dry runs.

```APIDOC
## huggingface_hub.file_download

### Description
Downloads a given file if it’s not already present in the local cache. The function supports specifying repository ID, filename, revision, and various caching and download options.

### Method
```python
def hf_hub_download(
    repo_id: str,
    filename: str,
    subfolder: str | None = None,
    repo_type: str | None = None,
    revision: str | None = None,
    library_name: str | None = None,
    library_version: str | None = None,
    cache_dir: str | pathlib.Path | None = None,
    local_dir: str | pathlib.Path | None = None,
    user_agent: dict | str | None = None,
    force_download: bool = False,
    etag_timeout: float = 10,
    token: bool | str | None = None,
    local_files_only: bool = False,
    headers: dict[str, str] | None = None,
    tqdm_class: type[tqdm.asyncio.tqdm_asyncio] | None = None,
    dry_run: bool = False
) -> str or DryRunFileInfo
```

### Parameters
#### Path Parameters
- **repo_id** (str) - A user or an organization name and a repo name separated by a `/`.
- **filename** (str) - The name of the file in the repo.

#### Query Parameters
- **subfolder** (str, optional) - An optional value corresponding to a folder inside the model repo.
- **repo_type** (str, optional) - Set to `"dataset"`, `"space"` or `"kernel"` if downloading from a dataset, space or kernel repo, `None` or `"model"` if downloading from a model. Default is `None`.
- **revision** (str, optional) - An optional Git revision id which can be a branch name, a tag, or a commit hash.
- **library_name** (str, optional) - The name of the library to which the object corresponds.
- **library_version** (str, optional) - The version of the library.
- **cache_dir** (str or Path, optional) - Path to the folder where cached files are stored.
- **local_dir** (str or Path, optional) - If provided, the downloaded file will be placed under this directory.
- **user_agent** (dict or str, optional) - The user-agent info in the form of a dictionary or a string.
- **force_download** (bool, optional, defaults to `False`) - Whether the file should be downloaded even if it already exists in the local cache.
- **etag_timeout** (float, optional, defaults to `10`) - When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`.
- **token** (str or bool, optional) - A token to be used for the download. If `True`, the token is read from the HuggingFace config folder. If a string, it’s used as the authentication token.
- **local_files_only** (bool, optional, defaults to `False`) - If `True`, avoid downloading the file and return the path to the local cached file if it exists.
- **headers** (dict, optional) - Additional headers to be sent with the request.
- **tqdm_class** (tqdm, optional) - If provided, overwrites the default behavior for the progress bar. Passed argument must inherit from `tqdm.auto.tqdm` or at least mimic its behavior. Defaults to the custom HF progress bar that can be disabled by setting `HF_HUB_DISABLE_PROGRESS_BARS` environment variable.
- **dry_run** (bool, optional, defaults to `False`) - If `True`, perform a dry run without actually downloading the file. Returns a DryRunFileInfo object containing information about what would be downloaded.

### Returns
`str` or `DryRunFileInfo`
  * If `dry_run=False`: Local path of file or if networking is off, last version of file cached on disk.
  * If `dry_run=True`: A DryRunFileInfo object containing download information.

### Raises
`RepositoryNotFoundError`, `RevisionNotFoundError`, `~utils.RemoteEntryNotFoundError`, `LocalEntryNotFoundError`, `EnvironmentError`, `OSError`, `ValueError`
  * `RepositoryNotFoundError` — If the repository to download from cannot be found. This may be because it doesn’t exist, or because it is set to `private` and you do not have access.
  * `RevisionNotFoundError` — If the revision to download from cannot be found.
  * `~utils.RemoteEntryNotFoundError` — If the file to download cannot be found.
  * `LocalEntryNotFoundError` — If network is disabled or unavailable and file is not found in cache.
  * `EnvironmentError` — If `token=True` but the token cannot be found.
  * `OSError` — If ETag cannot be determined.
  * `ValueError` — If some parameter value is invalid.
```

--------------------------------

### huggingface_hub.file_download

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Downloads a whole snapshot of a repository's files at a specified revision. It supports filtering files using allow_patterns and ignore_patterns, and can perform a dry run to preview downloads.

```APIDOC
## POST /hf_hub_download

### Description
Downloads a whole snapshot of a repo’s files at the specified revision. This is useful when you want all files from a repo, because you don’t know which ones you will need a priori. All files are nested inside a folder in order to keep their actual filename relative to that folder. You can also filter which files to download using `allow_patterns` and `ignore_patterns`.

### Method
POST

### Endpoint
/hf_hub_download

### Parameters
#### Path Parameters
- **repo_id** (str) - Required - A user or an organization name and a repo name separated by a `/`.
- **repo_type** (str) - Optional - Set to "dataset", "space" or "kernel" if downloading from a dataset, space or kernel repo, `None` or "model" if downloading from a model. Default is `None`.
- **revision** (str) - Optional - An optional Git revision id which can be a branch name, a tag, or a commit hash.
- **cache_dir** (str or Path) - Optional - Path to the folder where cached files are stored.
- **local_dir** (str or Path) - Optional - If provided, the downloaded files will be placed under this directory.
- **library_name** (str) - Optional - The name of the library to which the object corresponds.
- **library_version** (str) - Optional - The version of the library.
- **user_agent** (str or dict) - Optional - The user-agent info in the form of a dictionary or a string.
- **etag_timeout** (float) - Optional - When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `httpx.request`. Defaults to `10`.
- **force_download** (bool) - Optional - Whether the file should be downloaded even if it already exists in the local cache. Defaults to `False`.
- **token** (str or bool) - Optional - A token to be used for the download. If `True`, the token is read from the HuggingFace config folder. If a string, it’s used as the authentication token.
- **headers** (dict) - Optional - Additional headers to include in the request. Those headers take precedence over the others.
- **local_files_only** (bool) - Optional - If `True`, avoid downloading the file and return the path to the local cached file if it exists. Defaults to `False`.
- **allow_patterns** (list[str] or str) - Optional - If provided, only files matching at least one pattern are downloaded.
- **ignore_patterns** (list[str] or str) - Optional - If provided, files matching any of the patterns are not downloaded.
- **max_workers** (int) - Optional - Number of concurrent threads to download files (1 thread = 1 file download). Defaults to 8.
- **tqdm_class** (tqdm) - Optional - If provided, overwrites the default behavior for the progress bar. Passed argument must inherit from `tqdm.auto.tqdm` or at least mimic its behavior. Note that the `tqdm_class` is not passed to each individual download. Defaults to the custom HF progress bar that can be disabled by setting `HF_HUB_DISABLE_PROGRESS_BARS` environment variable.
- **dry_run** (bool) - Optional - If `True`, perform a dry run without actually downloading the files. Returns a list of DryRunFileInfo objects containing information about what would be downloaded. Defaults to `False`.

### Request Example
```json
{
  "repo_id": "bert-base-uncased",
  "revision": "main",
  "allow_patterns": "*.json"
}
```

### Response
#### Success Response (200)
- **str** or list of DryRunFileInfo - If `dry_run=False`: Local snapshot path. If `dry_run=True`: A list of DryRunFileInfo objects containing download information.

#### Response Example
```json
"/path/to/downloaded/files"
```

### Raises
- RepositoryNotFoundError: If the repository to download from cannot be found. This may be because it doesn’t exist, or because it is set to `private` and you do not have access.
- RevisionNotFoundError: If the revision to download from cannot be found.
- EnvironmentError: If `token=True` and the token cannot be found.
- OSError: if ETag cannot be determined.
- ValueError: if some parameter value is invalid.
```

--------------------------------

### hf_hub_download

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Downloads a single file from the Hugging Face Hub. This function handles caching and ensures you have the latest version of the file.

```APIDOC
## hf_hub_download

### Description
Downloads a single file from the Hugging Face Hub.

### Method
Not applicable (Python function)

### Endpoint
Not applicable (Python function)

### Parameters
#### Path Parameters
Not applicable

#### Query Parameters
Not applicable

#### Request Body
Not applicable

### Request Example
```python
from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="bert-base-uncased", filename="config.json")
```

### Response
#### Success Response
Returns the path to the downloaded file on the local filesystem.

#### Response Example
```
'/path/to/your/cache/bert-base-uncased/config.json'
```
```

--------------------------------

### hf_hub_url

Source: https://huggingface.co/docs/huggingface_hub/package_reference/file_download

Constructs the URL of a file from the given information. The resolved address can be a huggingface.co-hosted URL or a link to Cloudfront for large files.

```APIDOC
## hf_hub_url

### Description
Constructs the URL of a file from the given information. The resolved address can either be a huggingface.co-hosted url, or a link to Cloudfront (a Content Delivery Network, or CDN) for large files which are more than a few MBs.

### Method
Not applicable (Python function)

### Endpoint
Not applicable (Python function)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **repo_id** (`str`) - Required - A namespace (user or an organization) name and a repo name separated by a `/`.
* **filename** (`str`) - Required - The name of the file in the repo.
* **subfolder** (`str`) - Optional - An optional value corresponding to a folder inside the repo.
* **repo_type** (`str`) - Optional - Set to "dataset", "space" or "kernel" if downloading from a dataset, space or kernel repo, `None` or "model" if downloading from a model. Default is `None`.
* **revision** (`str`) - Optional - An optional Git revision id which can be a branch name, a tag, or a commit hash.

### Request Example
```python
>>> from huggingface_hub import hf_hub_url

>>> hf_hub_url(
...     repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin"
... )
'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin'
```

### Response
#### Success Response
Returns the URL string of the file.

#### Response Example
'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin'
```
Tokens:
Raw