FakeShield

https://github.com/zhipeixu/fakeshield
Admin
FakeShield is a novel multi-modal framework for explainable image forgery detection and localization...
Tokens:2,552
Snippets:33
Trust Score:7.9
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Download FakeShield weights

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Install the huggingface_hub library and download the model weights to the local weight directory.

```bash
pip install huggingface_hub
huggingface-cli download --resume-download zhipeixu/fakeshield-v1-22b --local-dir weight/
```

--------------------------------

### Install DTE-FDM

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Install the Domain Tag-guided Explainable Forgery Detection Module.

```bash
cd ../DTE-FDM
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```

--------------------------------

### Navigate and Install MMCV in Docker

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Commands to prepare the environment inside the Docker container.

```bash
cd /workspace/FakeShield
```

```bash
git clone https://github.com/open-mmlab/mmcv
```

--------------------------------

### Install Dependencies via Pip

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Install project requirements and MMCV library.

```bash
apt update && apt install git
pip install -r requirements.txt

## Install MMCV
git clone https://github.com/open-mmlab/mmcv
cd mmcv
git checkout v1.4.7
MMCV_WITH_OPS=1 pip install -e .
```

--------------------------------

### Download SAM weights

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Use wget to fetch the required SAM pre-training weights into the weight directory.

```bash
wget https://huggingface.co/ybelkada/segment-anything/resolve/main/checkpoints/sam_vit_h_4b8939.pth -P weight/
```

--------------------------------

### Run CLI demo

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Execute the demo script to process images using the configured environment variables.

```bash
bash scripts/cli_demo.sh
```

--------------------------------

### Clone Repository

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Initial step to download the FakeShield source code.

```bash
git clone https://github.com/zhipeixu/FakeShield.git
cd FakeShield
```

--------------------------------

### Run Docker Containers

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Launch the containers with GPU support.

```bash
docker run --gpus all -it --rm \
    -v $(pwd):/workspace/FakeShield \
    zhipeixu/dte-fdm:latest /bin/bash

docker run --gpus all -it --rm \
    -v $(pwd):/workspace/FakeShield \
    zhipeixu/mflm:latest /bin/bash
```

--------------------------------

### Dataset Directory Structure

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

The required file system hierarchy for organizing training and validation datasets.

```bash
dataset/
├── photoshop/                # PhotoShop Manipulation Dataset
│   ├── CASIAv2_Tp/           # CASIAv2 Tampered Images
│   │   ├── image/
│   │   └── mask/
│   ├── CASIAv2_Au/           # CASIAv2 Authentic Images
│   │   └── image/
│   ├── FR_Tp/                # Fantastic Reality Tampered Images
│   │   ├── image/
│   │   └── mask/
│   ├── FR_Au/                # Fantastic Reality Authentic Images
│   │   └── image/
│   ├── CASIAv1+_Tp/          # CASIAv1+ Tampered Images
│   │   ├── image/
│   │   └── mask/
│   ├── CASIAv1+_Au/          # CASIAv1+ Authentic Images
│   │   └── image/
│   ├── IMD2020_Tp/           # IMD2020 Tampered Images
│   │   ├── image/
│   │   └── mask/
│   ├── IMD2020_Au/           # IMD2020 Authentic Images
│   │   └── image/
│   ├── Columbia/             # Columbia Dataset
│   │   ├── image/
│   │   └── mask/
│   ├── coverage/             # Coverage Dataset
│   │   ├── image/
│   │   └── mask/
│   ├── NIST16/               # NIST16 Dataset
│   │   ├── image/
│   │   └── mask/
│   ├── DSO/                  # DSO Dataset
│   │   ├── image/
│   │   └── mask/
│   └── Korus/                # Korus Dataset
│       ├── image/
│       └── mask/
│
├── deepfake/                 # DeepFake Manipulation Dataset
│   ├── FaceAPP_Train/        # FaceAPP Training Data
│   │   ├── image/
│   │   └── mask/
│   ├── FaceAPP_Val/          # FaceAPP Validation Data
│   │   ├── image/
│   │   └── mask/
│   ├── FFHQ_Train/           # FFHQ Training Data
│   │   └── image/
│   └── FFHQ_Val/             # FFHQ Validation Data
│       └── image/
│
├── aigc/                     # AIGC Editing Manipulation Dataset
│   ├── SD_inpaint_Train/     # Stable Diffusion Inpainting Training Data
│   │   ├── image/
│   │   └── mask/
│   ├── SD_inpaint_Val/       # Stable Diffusion Inpainting Validation Data
│   │   ├── image/
│   │   └── mask/
│   ├── COCO2017_Train/       # COCO2017 Training Data
│   │   └── image/
│   └── COCO2017_Val/         # COCO2017 Validation Data
│       └── image/
│
└── MMTD_Set/                 # Multi-Modal Tamper Description Dataset
    └── MMTD-Set-34k.json     # JSON Training File
```

--------------------------------

### Fetch and Initialize Evaluation Data

Source: https://github.com/zhipeixu/fakeshield/blob/main/DTE-FDM/llava/eval/webpage/index.html

Retrieves evaluation data from a local JSON file and initializes the UI components for models and questions.

```javascript
// Fetch the JSON file
fetch('data.json')
.then(response => response.json())
.then(json_data => {
  // Populate the models and questions.
  populateModels(json_data.models);
  populateQuestions(json_data.questions);
  displayQuestion(currentQuestionIndex);
}).catch(error => console.error(error));
```

--------------------------------

### Pull Docker Images

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Download pre-built Docker images for the project modules.

```bash
docker pull zhipeixu/mflm:v1.0
docker pull zhipeixu/dte-fdm:v1.0
```

--------------------------------

### Weight directory structure

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

The expected file organization for the weight folder.

```text
 FakeShield/
    ├── weight/
    │   ├── fakeshield-v1-22b/
    │   │   ├── DTE-FDM/
    │   │   ├── MFLM/
    │   │   ├── DTG.pth
    │   ├── sam_vit_h_4b8939.pth
```

--------------------------------

### Run FakeShield evaluation script

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Executes the test script for the FakeShield model. Ensure environment variables are configured before running.

```bash
bash ./scripts/test.sh
```

--------------------------------

### Fine-tune MFLM with LoRA

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Execute the training script for MFLM. Customization is supported via environment variables including dataset selection and weight paths.

```bash
bash ./scripts/MFLM/finetune_lora.sh
```

--------------------------------

### Fine-tune DTE-FDM with LoRA

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

Execute the training script for DTE-FDM. Environment variables like OUTPUT_DIR, DATA_PATH, and WEIGHT_PATH can be used to customize the process.

```bash
bash ./scripts/DTE-FDM/finetune_lora.sh
```

--------------------------------

### Cite FakeShield

Source: https://github.com/zhipeixu/fakeshield/blob/main/README.md

BibTeX entry for referencing the FakeShield paper in academic work.

```bibtex
@inproceedings{xu2024fakeshield,
            title={FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models},
            author={Xu, Zhipei and Zhang, Xuanyu and Li, Runyi and Tang, Zecheng and Huang, Qing and Zhang, Jian},
            booktitle={International Conference on Learning Representations},
            year={2025}
    }
```
Tokens:
Raw