BiomedCLIP

https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
Admin
BiomedCLIP is a biomedical vision-language foundation model pretrained on 15 million biomedical...
Tokens:24,781
Snippets:136
Trust Score:10
Update:6 months ago
Show doc for...
Tokens:
Raw
### Load CLIP Vision Model

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/19

Instantiate the CLIPVisionModel from Hugging Face. Ensure the necessary libraries are installed.

```python
model = CLIPVisionModel.from_pretrained("microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224")
```

--------------------------------

### Load Model and Preprocessing Configurations

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blame/main/README.md

Load the model configuration and preprocessing settings from the downloaded JSON file. This setup is necessary before creating the model instance.

```python
model_name = "biomedclip_local"

with open("checkpoints/open_clip_config.json", "r") as f:
    config = json.load(f)
    model_cfg = config["model_cfg"]
    preprocess_cfg = config["preprocess_cfg"]


if (not model_name.startswith(HF_HUB_PREFIX)
    and model_name not in _MODEL_CONFIGS
    and config is not None):
    _MODEL_CONFIGS[model_name] = model_cfg
```

--------------------------------

### Create Conda Environment for BiomedCLIP

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blob/main/README.md

Sets up a Python 3.10 environment with necessary libraries for BiomedCLIP. Activate the environment before installing packages.

```bash
conda create -n biomedclip python=3.10 -y
conda activate biomedclip
pip install open_clip_torch==2.23.0 transformers==4.35.2 matplotlib
```

--------------------------------

### Load Model, Processor, and Get Image Features

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Load the CLIP model and processor, then fetch an image from a URL to extract image features. Ensure the processor is configured to return PyTorch tensors.

```python
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
```

```python
processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")
```

```python
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
```

```python
image = Image.open(requests.get(url, stream=True).raw)
```

```python
inputs = processor(images=image, return_tensors="pt")
```

```python
image_features = model.get_image_features(**inputs)
```

--------------------------------

### Load Tokenizer and Get Text Features

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Load the tokenizer and process text inputs to obtain text features. Ensure padding is enabled and tensors are returned.

```python
tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")
```

```python
inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
```

```python
text_features = model.get_text_features(**inputs)
```

--------------------------------

### Zero-Shot Image Classification with BiomedCLIP

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

Load the BiomedCLIP model and tokenizer from Hugging Face Hub. This example demonstrates zero-shot image classification by comparing image features with text labels using a predefined template.

```python
import torch
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer

# Load the model and config files from the Hugging Face Hub
model, preprocess = create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')


# Zero-shot image classification
template = 'this is a photo of '
labels = [
    'adenocarcinoma histopathology',
    'brain MRI',
    'covid line chart',
    'squamous cell carcinoma histopathology',
    'immunohistochemistry histopathology',
    'bone X-ray',
    'chest X-ray',
    'pie chart',
    'hematoxylin and eosin histopathology'
]

dataset_url = 'https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/'
test_imgs = [
    'squamous_cell_carcinoma_histopathology.jpeg',
    'H_and_E_histopathology.jpg',
    'bone_X-ray.jpg',
    'adenocarcinoma_histopathology.jpg',
    'covid_line_chart.png',
    'IHC_histopathology.jpg',
    'chest_X-ray.jpg',
    'brain_MRI.jpg',
    'pie_chart.png'
]
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
model.eval()

context_length = 256

images = torch.stack([preprocess(Image.open(urlopen(dataset_url + img))) for img in test_imgs]).to(device)
texts = tokenizer([template + l for l in labels], context_length=context_length).to(device)
with torch.no_grad():
    image_features, text_features, logit_scale = model(images, texts)

    logits = (logit_scale * image_features @ text_features.t()).detach().softmax(dim=-1)
    sorted_indices = torch.argsort(logits, dim=-1, descending=True)

    logits = logits.cpu().numpy()
    sorted_indices = sorted_indices.cpu().numpy()

top_k = -1

for i, img in enumerate(test_imgs):
    pred = labels[sorted_indices[i][0]]

    top_k = len(labels) if top_k == -1 else top_k
    print(img.split('/')[-1] + ':')
    for j in range(top_k):
        jth_index = sorted_indices[i][j]
        print(f'{labels[jth_index]}: {logits[i][jth_index]}')
    print('\n')

```

--------------------------------

### Zero-Shot Image Classification with BiomedCLIP

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blame/main/README.md

Load the BiomedCLIP model and tokenizer from Hugging Face Hub. This example demonstrates zero-shot image classification by processing a list of images against a set of candidate labels and printing the prediction results.

```python
import torch
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer

# Load the model and config files from the Hugging Face Hub
model, preprocess = create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')


# Zero-shot image classification
template = 'this is a photo of '
labels = [
    'adenocarcinoma histopathology',
    'brain MRI',
    'covid line chart',
    'squamous cell carcinoma histopathology',
    'immunohistochemistry histopathology',
    'bone X-ray',
    'chest X-ray',
    'pie chart',
    'hematoxylin and eosin histopathology'
]

dataset_url = 'https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/'
test_imgs = [
    'squamous_cell_carcinoma_histopathology.jpeg',
    'H_and_E_histopathology.jpg',
    'bone_X-ray.jpg',
    'adenocarcinoma_histopathology.jpg',
    'covid_line_chart.png',
    'IHC_histopathology.jpg',
    'chest_X-ray.jpg',
    'brain_MRI.jpg',
    'pie_chart.png'
]
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
model.eval()

context_length = 256

images = torch.stack([preprocess(Image.open(urlopen(dataset_url + img))) for img in test_imgs]).to(device)
texts = tokenizer([template + l for l in labels], context_length=context_length).to(device)
with torch.no_grad():
    image_features, text_features, logit_scale = model(images, texts)

    logits = (logit_scale * image_features @ text_features.t()).detach().softmax(dim=-1)
    sorted_indices = torch.argsort(logits, dim=-1, descending=True)

    logits = logits.cpu().numpy()
    sorted_indices = sorted_indices.cpu().numpy()

top_k = -1

for i, img in enumerate(test_imgs):
    pred = labels[sorted_indices[i][0]]

    top_k = len(labels) if top_k == -1 else top_k
    print(img.split('/')[-1] + ':')
    for j in range(top_k):
        jth_index = sorted_indices[i][j]
        print(f'{labels[jth_index]}: {logits[i][jth_index]}')
    print('\n')
```

--------------------------------

### Create Model and Transforms

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blame/main/README.md

Instantiate the model and associated image preprocessing transforms using the loaded configurations and downloaded model weights. Ensure the pretrained path points to the downloaded model file.

```python
tokenizer = get_tokenizer(model_name)

model, _, preprocess = create_model_and_transforms(
    model_name=model_name,
    pretrained="checkpoints/open_clip_pytorch_model.bin",
    **{f"image_{k}": v for k, v in preprocess_cfg.items()},
)
```

--------------------------------

### Get Image Features

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Extracts image features from the input image using the model's vision processing components.

```APIDOC
## GET /api/image/features

### Description
Extracts image features from the input image.

### Method
POST

### Endpoint
/api/image/features

### Parameters
#### Request Body
- **pixel_values** (torch.FloatTensor) - Required - Input pixel values.
- **output_attentions** (bool) - Optional - Whether to return attentions.
- **output_hidden_states** (bool) - Optional - Whether to return hidden states.
- **return_dict** (bool) - Optional - Whether to return a dictionary.

### Request Example
```json
{
  "pixel_values": [[[0.1, 0.2, ...], [0.3, 0.4, ...]]],
  "return_dict": true
}
```

### Response
#### Success Response (200)
- **image_features** (torch.FloatTensor) - The image embeddings.

#### Response Example
```json
{
  "image_features": [[0.4, 0.5, 0.6, ...]]
}
```
```

--------------------------------

### Get Text Features

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Extracts text features from the input text using the model's text processing components.

```APIDOC
## GET /api/text/features

### Description
Extracts text features from the input text.

### Method
POST

### Endpoint
/api/text/features

### Parameters
#### Request Body
- **input_ids** (torch.LongTensor) - Required - Input token IDs.
- **attention_mask** (torch.Tensor) - Optional - Attention mask for the input IDs.
- **token_type_ids** (torch.LongTensor) - Optional - Token type IDs.
- **position_ids** (torch.LongTensor) - Optional - Position IDs.
- **output_attentions** (bool) - Optional - Whether to return attentions.
- **output_hidden_states** (bool) - Optional - Whether to return hidden states.
- **return_dict** (bool) - Optional - Whether to return a dictionary.

### Request Example
```json
{
  "input_ids": [101, 2023, 2003, 1037, 3793, 2000, 1037, 4721, 102],
  "attention_mask": [1, 1, 1, 1, 1, 1, 1, 1, 1],
  "return_dict": true
}
```

### Response
#### Success Response (200)
- **text_features** (torch.FloatTensor) - The text embeddings.

#### Response Example
```json
{
  "text_features": [[0.1, 0.2, 0.3, ...]]
}
```
```

--------------------------------

### Load model from local files

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/README.md?download=true

Downloads model weights and configuration from the Hugging Face Hub to a local directory, then initializes the model for inference.

```python
import json

from urllib.request import urlopen
from PIL import Image
import torch
from huggingface_hub import hf_hub_download
from open_clip import create_model_and_transforms, get_tokenizer
from open_clip.factory import HF_HUB_PREFIX, _MODEL_CONFIGS


# Download the model and config files
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_pytorch_model.bin",
    local_dir="checkpoints"
)
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_config.json",
    local_dir="checkpoints"
)


# Load the model and config files
model_name = "biomedclip_local"

with open("checkpoints/open_clip_config.json", "r") as f:
    config = json.load(f)
    model_cfg = config["model_cfg"]
    preprocess_cfg = config["preprocess_cfg"]


if (not model_name.startswith(HF_HUB_PREFIX)
    and model_name not in _MODEL_CONFIGS
    and config is not None):
    _MODEL_CONFIGS[model_name] = model_cfg

tokenizer = get_tokenizer(model_name)

model, _, preprocess = create_model_and_transforms(
    model_name=model_name,
    pretrained="checkpoints/open_clip_pytorch_model.bin",
    **{f"image_{k}": v for k, v in preprocess_cfg.items()},
)


# Zero-shot image classification
template = 'this is a photo of '
labels = [
    'adenocarcinoma histopathology',
    'brain MRI',
    'covid line chart',
    'squamous cell carcinoma histopathology',
    'immunohistochemistry histopathology',
    'bone X-ray',
    'chest X-ray',
    'pie chart',
    'hematoxylin and eosin histopathology'
]

dataset_url = 'https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/'
test_imgs = [
    'squamous_cell_carcinoma_histopathology.jpeg',
    'H_and_E_histopathology.jpg',
    'bone_X-ray.jpg',
    'adenocarcinoma_histopathology.jpg',
    'covid_line_chart.png',
    'IHC_histopathology.jpg',
    'chest_X-ray.jpg',
    'brain_MRI.jpg',
    'pie_chart.png'
]
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
model.eval()

context_length = 256

images = torch.stack([preprocess(Image.open(urlopen(dataset_url + img))) for img in test_imgs]).to(device)
texts = tokenizer([template + l for l in labels], context_length=context_length).to(device)
with torch.no_grad():
    image_features, text_features, logit_scale = model(images, texts)

    logits = (logit_scale * image_features @ text_features.t()).detach().softmax(dim=-1)
    sorted_indices = torch.argsort(logits, dim=-1, descending=True)

    logits = logits.cpu().numpy()
    sorted_indices = sorted_indices.cpu().numpy()

top_k = -1

for i, img in enumerate(test_imgs):
    pred = labels[sorted_indices[i][0]]

    top_k = len(labels) if top_k == -1 else top_k
    print(img.split('/')[-1] + ':')
    for j in range(top_k):
        jth_index = sorted_indices[i][j]
        print(f'{labels[jth_index]}: {logits[i][jth_index]}')
    print('\n')
```

--------------------------------

### Download Model and Config Files

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blame/main/README.md

Use hf_hub_download to fetch the model weights and configuration files. Ensure the local_dir is set to where you want to store these files.

```python
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_pytorch_model.bin",
    local_dir="checkpoints"
)
```

```python
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_config.json",
    local_dir="checkpoints"
)
```

--------------------------------

### Load Model and Configuration

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

Loads the model and its configuration files. Ensure the 'checkpoints' directory exists and contains 'open_clip_config.json' and 'open_clip_pytorch_model.bin'.

```python
model_name = "biomedclip_local"

with open("checkpoints/open_clip_config.json", "r") as f:
    config = json.load(f)
    model_cfg = config["model_cfg"]
    preprocess_cfg = config["preprocess_cfg"]


if (not model_name.startswith(HF_HUB_PREFIX)
    and model_name not in _MODEL_CONFIGS
    and config is not None):
    _MODEL_CONFIGS[model_name] = model_cfg

tokenizer = get_tokenizer(model_name)

model, _, preprocess = create_model_and_transforms(
    model_name=model_name,
    pretrained="checkpoints/open_clip_pytorch_model.bin",
    **{f"image_{k}": v for k, v in preprocess_cfg.items()},
)
```

--------------------------------

### Initialize BiomedCLIPAttention

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIPAttention module, setting up linear projections for queries, keys, and values, as well as the output projection. It also includes error handling for embed_dim divisibility by num_heads.

```python
def __init__(self, config):
  super().__init__()
  self.config = config
  self.embed_dim = config.hidden_size
  self.num_heads = config.num_attention_heads
  self.head_dim = self.embed_dim // self.num_heads
  if self.head_dim * self.num_heads != self.embed_dim:
  raise ValueError(
  f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:
  f" {self.num_heads})."
  )
  self.scale = self.head_dim**-0.5
  self.dropout = nn.Dropout(config.attention_dropout)

  self.k_proj = nn.Linear(self.embed_dim, self.embed_dim)
  self.v_proj = nn.Linear(self.embed_dim, self.embed_dim)
  self.q_proj = nn.Linear(self.embed_dim, self.embed_dim)
  self.out_proj = nn.Linear(self.embed_dim, self.embed_dim)
```

--------------------------------

### BiomedCLIPProcessor Initialization

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIPProcessor with an image processor and a tokenizer. It raises a ValueError if either is not provided.

```APIDOC
## BiomedCLIPProcessor Initialization

### Description
Initializes the BiomedCLIPProcessor, requiring both an image processor and a tokenizer. It handles a deprecated `feature_extractor` argument and ensures that essential components are provided.

### Method
`__init__`

### Parameters
- **image_processor** (CLIPImageProcessor) - Required - The image processor to use.
- **tokenizer** (BertTokenizer or BertTokenizerFast) - Required - The tokenizer to use.
- **feature_extractor** (CLIPImageProcessor) - Deprecated - Use `image_processor` instead.
- **kwargs** - Additional keyword arguments to be passed to the image processor or tokenizer.
```

--------------------------------

### Prepare Text and Images for CLIP Model

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Prepares text and/or images for a CLIP model. Forwards text to the tokenizer and images to the image processor. Supports various return tensor types.

```python
def __call__(self, text=None, images=None, return_tensors=None, **kwargs):
    """
    Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text`
    and `kwargs` arguments to CLIPTokenizerFast's [`~CLIPTokenizerFast.__call__`] if `text` is not `None` to encode
    the text. To prepare the image(s), this method forwards the `images` and `kwrags` arguments to
    CLIPImageProcessor's [`~CLIPImageProcessor.__call__`] if `images` is not `None`. Please refer to the doctsring
    of the above two methods for more information.

    Args:
        text (`str`, `List[str]`, `List[List[str]]`):
            The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
            (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
            `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
        images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`):
            The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch
            tensor. Both channels-first and channels-last formats are supported.

        return_tensors (`str` or [`~utils.TensorType`], *optional*):
            If set, will return tensors of a particular framework. Acceptable values are:

            - `'tf'`: Return TensorFlow `tf.constant` objects.
            - `'pt'`: Return PyTorch `torch.Tensor` objects.
            - `'np'`: Return NumPy `np.ndarray` objects.
            - `'jax'`: Return JAX `jnp.ndarray` objects.

    Returns:
        [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:

            - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`.
            - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when
              `return_attention_mask=True` or if *
```

```python
attention_mask" in self.model_input_names and if `text` is not
              `None`).
            - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`.
    """
    tokenizer_kwargs, image_processor_kwargs = {}, {}
    if kwargs:
        tokenizer_kwargs = {k: v for k, v in kwargs.items() if k not in self.image_processor._valid_processor_keys}
        image_processor_kwargs = {
            k: v for k, v in kwargs.items() if k in self.image_processor._valid_processor_keys
        }

    if text is None and images is None:
        raise ValueError("You have to specify either text or images. Both cannot be none.")

    if text is not None:
        encoding = self.tokenizer(text, return_tensors=return_tensors, **tokenizer_kwargs)

    if images is not None:

```

--------------------------------

### Initialize BiomedCLIPProcessor

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIPProcessor with an image processor and a tokenizer. Raises a ValueError if either is not provided. Deprecated 'feature_extractor' argument is supported for backward compatibility.

```python
from transformers.processing_utils import ProcessorMixin
from transformers.tokenization_utils_base import BatchEncoding

class BiomedCLIPProcessor(ProcessorMixin):
    """
    Constructs a CLIP processor which wraps a CLIP image processor and a CLIP tokenizer into a single processor.

    [`CLIPProcessor`] offers all the functionalities of [`CLIPImageProcessor`] and [`CLIPTokenizerFast`]. See the
    [`~CLIPProcessor.__call__`] and [`~CLIPProcessor.decode`] for more information.

    Args:
        image_processor ([`CLIPImageProcessor`], *optional*):
            The image processor is a required input.
        tokenizer ([`CLIPTokenizerFast`], *optional*):
            The tokenizer is a required input.
    """

    attributes = ["image_processor", "tokenizer"]
    image_processor_class = "CLIPImageProcessor"
    tokenizer_class = ("BertTokenizer", "BertTokenizerFast")

    def __init__(self, image_processor=None, tokenizer=None, **kwargs):
        feature_extractor = None
        if "feature_extractor" in kwargs:
            warnings.warn(
                "The `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor`"
                " instead.",
                FutureWarning,
            )
            feature_extractor = kwargs.pop("feature_extractor")

        image_processor = image_processor if image_processor is not None else feature_extractor
        if image_processor is None:
            raise ValueError("You need to specify an `image_processor`.")
        if tokenizer is None:
            raise ValueError("You need to specify a `tokenizer`.")

        super().__init__(image_processor, tokenizer)
```

--------------------------------

### Download and Load BiomedCLIP Model Locally

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/25/files

Download model and config files from Hugging Face Hub and load them locally. This is useful for using the model without an internet connection after the initial download.

```python
import json
from urllib.request import urlopen
from PIL import Image
import torch
from huggingface_hub import hf_hub_download
from open_clip import create_model_and_transforms, get_tokenizer
from open_clip.factory import HF_HUB_PREFIX, _MODEL_CONFIGS

# Download the model and config files
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_pytorch_model.bin",
    local_dir="checkpoints"
)
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_config.json",
    local_dir="checkpoints"
)

# Load the model and config files
model_name = "biomedclip_local"

with open("checkpoints/open_clip_config.json", "r") as f:
    config = json.load(f)
model_cfg = config["model_cfg"]
preprocess_cfg = config["preprocess_cfg"]

if (not model_name.startswith(HF_HUB_PREFIX)
and model_name not in _MODEL_CONFIGS
and config is not None):
    _MODEL_CONFIGS[model_name] = model_cfg

tokenizer = get_tokenizer(model_name)

model, _, preprocess = create_model_and_transforms(
    model_name=model_name,
    pretrained="checkpoints/open_clip_pytorch_model.bin",
    **{f"image_{k}": v for k, v in preprocess_cfg.items()},
)
```

--------------------------------

### Git LFS pointer for model binary

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Git LFS pointer file content for the pytorch_model.bin file.

```text
version https://git-lfs.github.com/spec/v1
oid sha256:5bdc400de59a85620ddc7584d06913dc901c47f22647899c6addec71b9a5c9a2
size 783733062
```

--------------------------------

### Download BiomedCLIP Model Locally

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

Download the model and configuration files for BiomedCLIP from the Hugging Face Hub to a local 'checkpoints' directory. This allows for offline use or custom loading procedures.

```python
import json

from urllib.request import urlopen
from PIL import Image
import torch
from huggingface_hub import hf_hub_download
from open_clip import create_model_and_transforms, get_tokenizer
from open_clip.factory import HF_HUB_PREFIX, _MODEL_CONFIGS


# Download the model and config files
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_pytorch_model.bin",
    local_dir="checkpoints"
)
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_config.json",
    local_dir="checkpoints"
)



```

--------------------------------

### Model Input Names

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Returns a list of input names for both the tokenizer and image processor.

```APIDOC
## Model Input Names

### Description
Returns a combined list of input names from both the tokenizer and the image processor.

### Method
`model_input_names` (property)

### Response
- `list` of strings: The input names for the model.
```

--------------------------------

### CLIP Model Forward Pass with Text and Image Inputs

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Perform a forward pass of the CLIP model using both text and image inputs. The processor should handle text and image preprocessing, returning PyTorch tensors with padding enabled.

```python
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
```

```python
processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")
```

```python
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
```

```python
image = Image.open(requests.get(url, stream=True).raw)
```

```python
inputs = processor(
        text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
    )
```

--------------------------------

### Load BiomedCLIP Model from Local Files

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blob/main/README.md

Downloads and loads the BiomedCLIP model and configuration from Hugging Face Hub to local files. This is useful for offline use or when network access is limited. Ensure 'checkpoints' directory exists.

```python
import json

from urllib.request import urlopen
from PIL import Image
import torch
from huggingface_hub import hf_hub_download
from open_clip import create_model_and_transforms, get_tokenizer
from open_clip.factory import HF_HUB_PREFIX, _MODEL_CONFIGS


# Download the model and config files
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_pytorch_model.bin",
    local_dir="checkpoints"
)
hf_hub_download(
    repo_id="microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224",
    filename="open_clip_config.json",
    local_dir="checkpoints"
)


# Load the model and config files
model_name = "biomedclip_local"

with open("checkpoints/open_clip_config.json", "r") as f:
    config = json.load(f)
    model_cfg = config["model_cfg"]
    preprocess_cfg = config["preprocess_cfg"]


if (not model_name.startswith(HF_HUB_PREFIX)
    and model_name not in _MODEL_CONFIGS
    and config is not None):
    _MODEL_CONFIGS[model_name] = model_cfg

tokenizer = get_tokenizer(model_name)

model, _, preprocess = create_model_and_transforms(
    model_name=model_name,
    pretrained="checkpoints/open_clip_pytorch_model.bin",
    **{f"image_{k}": v for k, v in preprocess_cfg.items()},
)

```

--------------------------------

### BiomedCLIP Model Initialization and Feature Extraction

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIP model with vision and text transformers and provides methods for extracting text features.

```python
class BiomedCLIPModel(CLIPPreTrainedModel):  
config_class = BiomedCLIPConfig  
_no_split_modules = ["BiomedCLIPTextEmbeddings", "BiomedCLIPEncoderLayer"]  
  
def __init__(self, config: BiomedCLIPConfig):  
super().__init__(config)  
  
if not isinstance(config.text_config, CLIPTextConfig):  
raise ValueError(  
"config.text_config is expected to be of type CLIPTextConfig but is of type"  
f" {type(config.text_config)}."  
)  
  
if not isinstance(config.vision_config, CLIPVisionConfig):  
raise ValueError(  
"config.vision_config is expected to be of type CLIPVisionConfig but is of type"  
f" {type(config.vision_config)}."  
)  
  
text_config = config.text_config  
text_projection_config = config.text_projection_config  
vision_config = config.vision_config  
 
  
self.projection_dim = config.projection_dim  
self.text_embed_dim = text_config.hidden_size  
self.vision_embed_dim = vision_config.hidden_size  
  
self.text_model = BiomedCLIPTextTransformer(text_config)  
self.vision_model = BiomedCLIPVisionTransformer(vision_config)  
 
self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False)  
 
self.text_projection = BiomedCLIPTextProjection(text_projection_config)  
 
self.logit_scale = nn.Parameter(torch.tensor(self.config.logit_scale_init_value))  
  
# Initialize weights and apply final processing  
self.post_init()  
  
def get_text_features(  
self,  
input_ids: Optional[torch.Tensor] = None,  
attention_mask: Optional[torch.Tensor] = None,  
token_type_ids: Optional[torch.Tensor] = None,  
position_ids: Optional[torch.Tensor] = None,  
output_attentions: Optional[bool] = None,  
output_hidden_states: Optional[bool] = None,  
return_dict: Optional[bool] = None,  
) -> torch.FloatTensor:  
r"""  
Returns:  
text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by  
applying the projection layer to the pooled output of [`CLIPTextModel`].  
  
Examples:  
  
```python  
>>> from transformers import AutoTokenizer, CLIPModel  
  
>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
```

--------------------------------

### BiomedCLIP Image Classification Model Initialization

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIPForImageClassification model, setting up the vision transformer and a linear classifier head. The classifier is an identity function if num_labels is not specified.

```python
class BiomedCLIPForImageClassification(CLIPPreTrainedModel):
    main_input_name = "pixel_values"

    def __init__(self, config: BiomedCLIPConfig) -> None:
        super().__init__(config)

        self.num_labels = config.num_labels
        self.vision_model = BiomedCLIPVisionTransformer(config.vision_config)

        # Classifier head
        self.classifier = (
            nn.Linear(config.vision_config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()
        )

        # Initialize weights and apply final processing
        self.post_init()
```

--------------------------------

### Perform Zero-Shot Image Classification

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blob/main/README.md?code=true

This snippet demonstrates how to process images and text labels to perform zero-shot classification using a pre-loaded model.

```python
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')  
model.to(device)  
model.eval()  
  
context_length = 256  
  
images = torch.stack([preprocess(Image.open(urlopen(dataset_url + img))) for img in test_imgs]).to(device)  
texts = tokenizer([template + l for l in labels], context_length=context_length).to(device)  
with torch.no_grad():  
 image_features, text_features, logit_scale = model(images, texts)  
  
 logits = (logit_scale * image_features @ text_features.t()).detach().softmax(dim=-1)  
 sorted_indices = torch.argsort(logits, dim=-1, descending=True)  
  
 logits = logits.cpu().numpy()  
 sorted_indices = sorted_indices.cpu().numpy()  
  
 top_k = -1  
  
 for i, img in enumerate(test_imgs):  
  pred = labels[sorted_indices[i][0]]  
  
  top_k = len(labels) if top_k == -1 else top_k  
  print(img.split('/')[-1] + ':')  
  for j in range(top_k):  
  jth_index = sorted_indices[i][j]  
  print(f'{labels[jth_index]}: {logits[i][jth_index]}')  
  print('\n')
```

--------------------------------

### Load BiomedCLIP Model from Hugging Face Hub

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blob/main/README.md

Loads the BiomedCLIP model and tokenizer from the Hugging Face Hub using open_clip. This is the first step to using the model for vision-language tasks.

```python
import torch
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer

# Load the model and config files from the Hugging Face Hub
model, preprocess = create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
```

--------------------------------

### Import necessary libraries for PyTorch and Transformers

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Imports essential modules from PyTorch and HuggingFace's transformers library, including dataclasses, typing utilities, math functions, neural network modules, and various model output classes. These are foundational for building and utilizing the BiomedCLIP model.

```python
from dataclasses import dataclass
from typing import Any, Optional, Tuple, Union, List

import math
import torch
import torch.utils.checkpoint
from torch import nn
from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss

from transformers.activations import ACT2FN
from transformers.modeling_attn_mask_utils import _create_4d_causal_attention_mask, _prepare_4d_attention_mask
from transformers.modeling_outputs import (
    BaseModelOutput,
    BaseModelOutputWithPooling,
    ImageClassifierOutput,
    BaseModelOutputWithPoolingAndCrossAttentions,
    BaseModelOutputWithPastAndCrossAttentions
)
from transformers.modeling_utils import PreTrainedModel
from transformers.utils import (
    ModelOutput,
    add_code_sample_docstrings,
    add_start_docstrings,
    add_start_docstrings_to_model_forward,
    logging,
    replace_return_docstrings,
)
```

--------------------------------

### Retrieve model input names

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Combines input names from both the tokenizer and the image processor.

```python
@property
def model_input_names(self):
  tokenizer_input_names = self.tokenizer.model_input_names
  image_processor_input_names = self.image_processor.model_input_names
  return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
```

--------------------------------

### Import Model with Remote Code

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20

Import the model directly using `from_pretrained` by setting `trust_remote_code=True`. This is necessary for models that include custom code.

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224", trust_remote_code=True)
```

--------------------------------

### Load BiomedCLIP Model from Local Files

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blame/main/README.md

This snippet shows how to load the BiomedCLIP model and associated transforms from local files. It imports necessary libraries including torch, PIL, and huggingface_hub.

```python
import json

from urllib.request import urlopen
from PIL import Image
import torch
from huggingface_hub import hf_hub_download
from open_clip import create_model_and_transforms, get_tokenizer
from open_clip.factory import HF_HUB_PREFIX, _MODEL_CONFIGS

```

--------------------------------

### Initialize BiomedCLIP Vision Transformer

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes the BiomedCLIP Vision Transformer. It sets up the configuration, embedding dimension, and the vision embedding layer.

```python
class BiomedCLIPVisionTransformer(nn.Module):
    def __init__(self, config: CLIPVisionConfig):
        super().__init__()
        self.config = config
        embed_dim = config.hidden_size

        self.embeddings = BiomedCLIPVisionEmbeddings(config)
```

--------------------------------

### BiomedCLIPProcessor Call Method

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Prepares text and/or images for the BiomedCLIP model. It forwards text to the tokenizer and images to the image processor, returning encoded data.

```APIDOC
## POST /prepare_data

### Description
This method prepares text and/or images for the BiomedCLIP model. It tokenizes the input text and processes the input images using the respective components of the processor. It can return tensors in different frameworks like TensorFlow, PyTorch, NumPy, or JAX.

### Method
`__call__`

### Endpoint
N/A (This is a class method, not a direct API endpoint)

### Parameters
#### Path Parameters
None

#### Query Parameters
- **return_tensors** (str) - Optional - Specifies the framework for the returned tensors ('tf', 'pt', 'np', 'jax').

#### Request Body
- **text** (str or List[str] or List[List[str]]) - Optional - The text sequence(s) to encode.
- **images** (PIL.Image.Image or np.ndarray or torch.Tensor or List[PIL.Image.Image] or List[np.ndarray] or List[torch.Tensor]) - Optional - The image(s) to process.
- ****kwargs** - Additional keyword arguments to be passed to the tokenizer or image processor.

### Request Example
```json
{
  "text": "A picture of a cat",
  "images": "path/to/image.jpg",
  "return_tensors": "pt"
}
```

### Response
#### Success Response (200)
- **input_ids** (List[int]) - Token IDs for the input text. Returned when `text` is not None.
- **attention_mask** (List[int]) - Attention mask for the input text. Returned when `text` is not None and `return_attention_mask` is True or implied.
- **pixel_values** (List or np.ndarray or torch.Tensor) - Processed pixel values for the input image. Returned when `images` is not None.

#### Response Example
```json
{
  "input_ids": [101, 2057, 2003, 2794, 102],
  "attention_mask": [1, 1, 1, 1, 1],
  "pixel_values": [
    0.123, 0.456, ...
  ]
}
```
```

--------------------------------

### Batch decode and decode methods

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Wrappers for the underlying tokenizer's decoding methods.

```python
def batch_decode(self, *args, **kwargs):
  """
  This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please
  refer to the docstring of this method for more information.
  """
  return self.tokenizer.batch_decode(*args, **kwargs)
```

```python
def decode(self, *args, **kwargs):
  """
  This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to
  the docstring of this method for more information.
  """
  return self.tokenizer.decode(*args, **kwargs)
```

--------------------------------

### Load BiomedCLIP Tokenizer

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/12

Use this to load the BiomedCLIP tokenizer from Hugging Face Hub. This tokenizer is compatible with open_clip.

```python
tokenizer = open_clip.get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
```

--------------------------------

### Load BiomedCLIP Model with Hugging Face Transformers

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/10

Use this code to load the BiomedCLIP model and processor directly using Hugging Face's AutoModel and AutoProcessor classes. Ensure the CLIP_DIR variable points to the model's directory.

```python
CLIP_DIR = "/path/to/BiomedCLIP"
model = AutoModel.from_pretrained(CLIP_DIR, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(CLIP_DIR, trust_remote_code=True)
```

--------------------------------

### Load and use BiomedCLIP for zero-shot classification

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/blob/main/README.md?code=true

Python code to load the model from the Hugging Face Hub and prepare labels for zero-shot image classification.

```python
import torch
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer

# Load the model and config files from the Hugging Face Hub
model, preprocess = create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')


# Zero-shot image classification
template = 'this is a photo of '
labels = [
 'adenocarcinoma histopathology',
 'brain MRI',
 'covid line chart',
 'squamous cell carcinoma histopathology',
 'immunohistochemistry histopathology',
 'bone X-ray',
 'chest X-ray',
 'pie chart',
 'hematoxylin and eosin histopathology'
]

dataset_url = 'https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/'
test_imgs = [
 'squamous_cell_carcinoma_histopathology.jpeg',
 'H_and_E_histopathology.jpg',
 'bone_X-ray.jpg',
 'adenocarcinoma_histopathology.jpg',
 'covid_line_chart.png',
 'IHC_histopathology.jpg',
 'chest_X-ray.jpg',
 'brain_MRI.jpg',
 'pie_chart.png'
]
```

--------------------------------

### Decoding

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Forwards arguments to the tokenizer's decode method.

```APIDOC
## Decoding

### Description
This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information.

### Method
`decode`

### Parameters
Refer to `CLIPTokenizerFast.decode` documentation for detailed parameters.
```

--------------------------------

### Initialize BiomedCLIPEncoderLayer

Source: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/discussions/20/files

Initializes a single encoder layer for the BiomedCLIP model. It sets up layer normalization and the self-attention and MLP modules. Supports both pre-norm and post-norm configurations.

```python
class BiomedCLIPEncoderLayer(nn.Module):
  def __init__(self, config: BiomedCLIPConfig, norm='pre'):
  super().__init__()
  self.embed_dim = config.hidden_size
  # pre-norm
  self.layer_norm1 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
  self.self_attn = BiomedCLIPAttention(config)
  self.layer_norm2 = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_eps)
  self.mlp = CLIPMLP(config)
  self.norm = norm

  if self.norm == 'pre':
  self.forward = self.pre_norm_forward
  elif self.norm == 'post':
  self.forward = self.post_norm_forward
```
Tokens:
Raw