MMDFND: Multi-modal Multi-Domain Fake News Detection

https://github.com/yutchina/mmdfnd
Admin
MMDFND is a multi-modal machine learning framework for detecting fake news across multiple domains...
Tokens:8,831
Snippets:39
Trust Score:5.2
Update:6 months ago
Show doc for...
Tokens:
Raw
### MMDFND Training Command Line Examples (Bash)

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Provides example command-line arguments for running the MMDFND training script. These examples show how to specify model name, dataset, epochs, batch size, learning rate, GPU device, and other training parameters.

```bash
# Command line usage
python main.py \
    --model_name MMDFND \
    --dataset weibo21 \
    --epoch 50 \
    --batchsize 64 \
    --lr 0.0001 \
    --gpu 0 \
    --early_stop 3 \
    --seed 3074

# For Weibo dataset
python main.py \
    --model_name MMDFND \
    --dataset weibo \
    --root_path ./data/ \
    --epoch 50 \
    --batchsize 64
```

--------------------------------

### Initialize and Run Training Pipeline with Run Class in Python

Source: https://context7.com/yutchina/mmdfnd/llms.txt

This Python code snippet shows how to initialize the `Run` class with the previously defined configuration and start the fake news detection training pipeline. It also includes example domain category dictionaries for both Weibo and Weibo21 datasets.

```python
from run import Run

# Initialize the Run class with configuration
runner = Run(config=config)

# Domain categories for Weibo dataset (9 domains)
weibo_category_dict = {
    "经济": 0,  # Economy
    "健康": 1,  # Health
    "军事": 2,  # Military
    "科学": 3,  # Science
    "政治": 4,  # Politics
    "国际": 5,  # International
    "教育": 6,  # Education
    "娱乐": 7,  # Entertainment
    "社会": 8   # Society
}

# Domain categories for Weibo21 dataset (9 domains)
weibo21_category_dict = {
    "科技": 0,      # Technology
    "军事": 1,      # Military
    "教育考试": 2,  # Education/Exams
    "灾难事故": 3,  # Disasters
    "政治": 4,      # Politics
    "医药健康": 5,  # Medical/Health
    "财经商业": 6,  # Finance/Business
    "文体娱乐": 7,  # Culture/Entertainment
    "社会生活": 8   # Social Life
}

# Start training
runner.main()
```

--------------------------------

### Initialize MultiDomainPLEFENDModel Architecture

Source: https://context7.com/yutchina/mmdfnd/llms.txt

This snippet defines the constructor for the MultiDomainPLEFENDModel, setting up frozen BERT and MAE encoders, CLIP integration, and the expert network routing structure. It initializes domain-specific and shared experts alongside gating networks for multi-domain feature fusion.

```python
import torch
import torch.nn as nn
from transformers import BertModel
import models_mae
from cn_clip.clip import load_from_name

class MultiDomainPLEFENDModel(torch.nn.Module):
    def __init__(self, emb_dim, mlp_dims, bert, out_channels, dropout):
        super(MultiDomainPLEFENDModel, self).__init__()
        self.num_expert = 6
        self.domain_num = 9
        self.unified_dim = emb_dim
        self.text_dim = 768
        self.image_dim = 768
        self.bert = BertModel.from_pretrained(bert).requires_grad_(False)
        self.image_model = models_mae.__dict__["mae_vit_base_patch16"](norm_pix_loss=False)
        checkpoint = torch.load('./mae_pretrain_vit_base.pth', map_location='cpu')
        self.image_model.load_state_dict(checkpoint['model'], strict=False)
        for param in self.image_model.parameters():
            param.requires_grad = False
        self.ClipModel, _ = load_from_name("ViT-B-16", device="cuda", download_root='./')
        self.domain_embedder = nn.Embedding(num_embeddings=self.domain_num, embedding_dim=emb_dim)
        feature_kernel = {1: 64, 2: 64, 3: 64, 5: 64, 10: 64}
        self.text_experts = nn.ModuleList([nn.ModuleList([cnn_extractor(emb_dim, feature_kernel) for _ in range(self.num_expert)]) for _ in range(self.domain_num)])
        self.image_experts = nn.ModuleList([nn.ModuleList([cnn_extractor(self.image_dim, feature_kernel) for _ in range(self.num_expert)]) for _ in range(self.domain_num)])
        self.text_share_expert = nn.ModuleList([nn.ModuleList([cnn_extractor(emb_dim, feature_kernel) for _ in range(self.num_expert * 2)])])
        self.text_gate_list = nn.ModuleList([nn.Sequential(nn.Linear(self.unified_dim * 2, self.unified_dim), nn.SiLU(), nn.Linear(self.unified_dim, self.num_expert * 3), nn.Dropout(0.1), nn.Softmax(dim=1)) for _ in range(self.domain_num)])
        self.final_classifier_list = nn.ModuleList([MLP(320, mlp_dims, dropout) for _ in range(self.domain_num)])
```

--------------------------------

### MMDFND Training Script Configuration and Execution (Python)

Source: https://context7.com/yutchina/mmdfnd/llms.txt

This script demonstrates how to train the MMDFND model. It includes argument parsing for hyperparameters, setting up CUDA devices, initializing random seeds for reproducibility, and configuring the model parameters before initiating the training process via the Run class.

```python
# main.py - Complete training script
import os
import argparse
import torch
import numpy as np
import random

# Parse arguments
parser = argparse.ArgumentParser()
parser.add_argument('--model_name', default='MMDFND')
parser.add_argument('--dataset', default='weibo21')
parser.add_argument('--epoch', type=int, default=50)
parser.add_argument('--batchsize', type=int, default=64)
parser.add_argument('--seed', type=int, default=3074)
parser.add_argument('--gpu', default='0')
parser.add_argument('--lr', type=float, default=0.0001)
args = parser.parse_args()

# Set GPU
os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu

# Set random seeds for reproducibility
seed = args.seed
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

from run import Run

# Build configuration
config = {
    'use_cuda': True,
    'batchsize': args.batchsize,
    'max_len': 197,
    'early_stop': 3,
    'num_workers': 4,
    'vocab_file': './pretrained_model/chinese_roberta_wwm_base_ext_pytorch/vocab.txt',
    'emb_type': 'bert',
    'bert': './pretrained_model/chinese_roberta_wwm_base_ext_pytorch',
    'root_path': './data/' if args.dataset == 'weibo' else './Weibo_21/',
    'weight_decay': 5e-5,
    'model': {'mlp': {'dims': [384], 'dropout': 0.2}},
    'emb_dim': 768,
    'lr': args.lr,
    'epoch': args.epoch,
    'model_name': args.model_name,
    'seed': args.seed,
    'save_param_dir': './param_model',
    'dataset': args.dataset
}

# Train model
if __name__ == '__main__':
    Run(config=config).main()
```

--------------------------------

### Implement Training Loop with Trainer Class

Source: https://context7.com/yutchina/mmdfnd/llms.txt

This class manages the training process for the MultiDomainPLEFENDModel. It includes logic for multi-task loss calculation, GPU data transfer, and automatic model saving based on early stopping criteria.

```python
import os
import tqdm
import torch
from utils.utils import clipdata2gpu, Averager, Recorder, metricsTrueFalse

class Trainer:
    def __init__(self, emb_dim, mlp_dims, bert, use_cuda, lr, dropout,
                 train_loader, val_loader, test_loader, category_dict,
                 weight_decay, save_param_dir, loss_weight=[1, 0.006, 0.009, 5e-5],
                 early_stop=5, epoches=100):
        self.lr = lr
        self.weight_decay = weight_decay
        self.train_loader = train_loader
        self.val_loader = val_loader
        self.test_loader = test_loader
        self.early_stop = early_stop
        self.epoches = epoches
        self.category_dict = category_dict
        self.use_cuda = use_cuda

        if not os.path.exists(save_param_dir):
            os.makedirs(save_param_dir)
        self.save_param_dir = save_param_dir

    def train(self):
        self.model = MultiDomainPLEFENDModel(
            self.emb_dim, self.mlp_dims, self.bert, 320, self.dropout
        )

        if self.use_cuda:
            self.model = self.model.cuda()

        loss_fn = torch.nn.BCELoss()
        optimizer = torch.optim.Adam(
            params=self.model.parameters(),
            lr=self.lr,
            weight_decay=self.weight_decay
        )
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.98)
        recorder = Recorder(self.early_stop)

        for epoch in range(self.epoches):
            self.model.train()
            train_data_iter = tqdm.tqdm(self.train_loader)
            avg_loss = Averager()

            for step_n, batch in enumerate(train_data_iter):
                batch_data = clipdata2gpu(batch)
                label = batch_data['label']
                category = batch_data['category']

                idxs = torch.tensor([index for index in category]).view(-1, 1)
                batch_label = torch.cat([
                    label[idxs.squeeze() == i] for i in range(9)
                ])

                final_pred, fusion_pred, image_pred, text_pred = self.model(**batch_data)

                loss = (0.7 * loss_fn(final_pred, batch_label.float()) +
                       0.1 * loss_fn(fusion_pred, batch_label.float()) +
                       0.1 * loss_fn(image_pred, batch_label.float()) +
                       0.1 * loss_fn(text_pred, batch_label.float()))

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                scheduler.step()
                avg_loss.add(loss.item())

            print(f'Training Epoch {epoch + 1}; Loss {avg_loss.item()}')

            results = self.test(self.val_loader)
            mark = recorder.add(results[0])

            if mark == 'save':
                torch.save(self.model.state_dict(),
                          os.path.join(self.save_param_dir, 'parameter_mmdfnd.pkl'))
            elif mark == 'esc':
                break

        self.model.load_state_dict(
            torch.load(os.path.join(self.save_param_dir, 'parameter_mmdfnd.pkl'))
        )
        final_results = self.test(self.test_loader)
        print(final_results[0])

        return final_results[0]
```

--------------------------------

### Configure Training Arguments with argparse in Python

Source: https://context7.com/yutchina/mmdfnd/llms.txt

This Python code snippet demonstrates how to configure training parameters for the MMDFND framework using the argparse module. It defines various command-line arguments for model selection, dataset, hyperparameters, and pretrained model paths, then organizes them into a configuration dictionary.

```python
import argparse
import os

parser = argparse.ArgumentParser()
parser.add_argument('--model_name', default='MMDFND')
parser.add_argument('--dataset', default='weibo21')  # Options: 'weibo21' or 'weibo'
parser.add_argument('--epoch', type=int, default=50)
parser.add_argument('--max_len', type=int, default=197)
parser.add_argument('--num_workers', type=int, default=4)
parser.add_argument('--early_stop', type=int, default=3)
parser.add_argument('--bert_vocab_file', default='./pretrained_model/chinese_roberta_wwm_base_ext_pytorch/vocab.txt')
parser.add_argument('--root_path', default='./data/')
parser.add_argument('--bert', default='./pretrained_model/chinese_roberta_wwm_base_ext_pytorch')
parser.add_argument('--batchsize', type=int, default=64)
parser.add_argument('--seed', type=int, default=3074)
parser.add_argument('--gpu', default='0')
parser.add_argument('--bert_emb_dim', type=int, default=768)
parser.add_argument('--lr', type=float, default=0.0001)
parser.add_argument('--save_param_dir', default='./param_model')
args = parser.parse_args()

# Configuration dictionary for the trainer
config = {
    'use_cuda': True,
    'batchsize': args.batchsize,
    'max_len': args.max_len,
    'early_stop': args.early_stop,
    'num_workers': args.num_workers,
    'vocab_file': args.bert_vocab_file,
    'emb_type': 'bert',
    'bert': args.bert,
    'root_path': args.root_path,
    'weight_decay': 5e-5,
    'model': {'mlp': {'dims': [384], 'dropout': 0.2}},
    'emb_dim': args.bert_emb_dim,
    'lr': args.lr,
    'epoch': args.epoch,
    'model_name': args.model_name,
    'seed': args.seed,
    'save_param_dir': args.save_param_dir,
    'dataset': args.dataset
}
```

--------------------------------

### Extract Image Features with ResNet and CLIP

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Functions to load, resize, and normalize images from local directories. It supports standard ResNet transformations and specific preprocessing for Chinese-CLIP models.

```python
import pickle
import torch
import os
from PIL import Image
from torchvision import transforms
import cn_clip.clip as clip
from cn_clip.clip import load_from_name

def read_image():
    image_list = {}
    file_list = ['data/nonrumor_images/', 'data/rumor_images/']
    data_transforms = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
    for path in file_list:
        for filename in os.listdir(path):
            try:
                im = Image.open(path + filename).convert('RGB')
                im = data_transforms(im)
                image_list[filename.split('/')[-1].split(".")[0].lower()] = im
            except Exception:
                print(f"Failed to load: {filename}")
    return image_list

def read_clip_image():
    image_list = {}
    file_list = ['data/nonrumor_images/', 'data/rumor_images/']
    device = "cuda" if torch.cuda.is_available() else "cpu"
    model, preprocess = load_from_name("ViT-B-16", device=device, download_root='./')
    for path in file_list:
        for filename in os.listdir(path):
            try:
                im = Image.open(path + filename)
                im = preprocess(im).unsqueeze(0).to(device)
                image_list[filename.split('/')[-1].split(".")[0].lower()] = im
            except Exception:
                print(f"Failed to load: {filename}")
    return image_list
```

--------------------------------

### Transfer Batch Data to GPU for Basic Model (Python)

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Moves batch data to the CUDA-enabled GPU for a basic model that does not utilize CLIP features. Similar to clipdata2gpu, it takes a tuple of tensors and returns a dictionary with tensors moved to the GPU.

```python
import torch

def data2gpu(batch):
    """Transfer batch data to GPU for basic model (without CLIP)."""
    batch_data = {
        'content': batch[0].cuda(),
        'content_masks': batch[1].cuda(),
        'label': batch[2].cuda(),
        'category': batch[3].cuda(),
        'image': batch[4].cuda()
    }
    return batch_data
```

--------------------------------

### Create BERT DataLoaders with Multimodal Features (Python)

Source: https://context7.com/yutchina/mmdfnd/llms.txt

The `bert_data` class constructs PyTorch DataLoaders for multimodal BERT tasks. It processes text content, labels, categories, and image features from specified file paths. The `load_data` method returns a DataLoader yielding token IDs, attention masks, labels, categories, MAE image features, CLIP image features, and CLIP text features.

```python
import pandas as pd
import pickle
from torch.utils.data import TensorDataset, DataLoader
import cn_clip.clip as clip
from cn_clip.clip import load_from_name

class bert_data:
    def __init__(self, max_len, batch_size, vocab_file, category_dict, num_workers=2):
        self.max_len = max_len
        self.batch_size = batch_size
        self.num_workers = num_workers
        self.vocab_file = vocab_file
        self.category_dict = category_dict

    def load_data(self, path, imagepath, clipimagepath, shuffle, text_only=False):
        """
        Load and prepare dataset with text, images, and CLIP features.

        Args:
            path: CSV file path with columns ['content', 'label', 'category']
            imagepath: Path to pickled MAE image features
            clipimagepath: Path to pickled CLIP image features
            shuffle: Whether to shuffle the data

        Returns:
            DataLoader with (token_ids, masks, label, category, image, clip_image, clip_text)
        """
        data = pd.read_csv(path, encoding='utf-8')

        content = data['content'].astype('object').to_numpy()
        label = torch.tensor(data['label'].astype(int).to_numpy())
        category = torch.tensor(
            data['category'].apply(lambda c: self.category_dict[c]).to_numpy()
        )

        token_ids, masks = word2input(content, self.vocab_file, self.max_len)
        ordered_image = pickle.load(open(imagepath, 'rb'))
        clip_image = pickle.load(open(clipimagepath, 'rb'))
        clip_text = clip.tokenize(content)

        dataset = TensorDataset(
            token_ids,      # BERT token IDs
            masks,          # Attention masks
            label,          # 0=real, 1=fake
            category,       # Domain category index
            ordered_image,  # MAE preprocessed images
            clip_image,     # CLIP preprocessed images
            clip_text       # CLIP tokenized text
        )

        dataloader = DataLoader(
            dataset=dataset,
            batch_size=self.batch_size,
            num_workers=self.num_workers,
            pin_memory=True,
            shuffle=shuffle
        )

        return dataloader

# Example usage
category_dict = {"经济": 0, "健康": 1, "军事": 2, "科学": 3, "政治": 4,
                 "国际": 5, "教育": 6, "娱乐": 7, "社会": 8}

loader = bert_data(
    max_len=197,
    batch_size=64,
    vocab_file='./pretrained_model/chinese_roberta_wwm_base_ext_pytorch/vocab.txt',
    category_dict=category_dict,
    num_workers=4
)

train_loader = loader.load_data(
    'data/train_origin.csv',
    'data/train_loader.pkl',
    'data/train_clip_loader.pkl',
    shuffle=True
)

```

--------------------------------

### Transfer Batch Data to GPU for CLIP Model (Python)

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Moves batch data, including images and text tokens, to the CUDA-enabled GPU for use with a CLIP-enhanced model. It expects a tuple of tensors from a DataLoader and returns a dictionary of tensors ready for model input.

```python
import torch

def clipdata2gpu(batch):
    """
    Transfer batch data to GPU for CLIP-enhanced model.

    Args:
        batch: Tuple of tensors from DataLoader

    Returns:
        Dictionary with all tensors moved to CUDA
    """
    batch_data = {
        'content': batch[0].cuda(),        # BERT token IDs [batch, seq_len]
        'content_masks': batch[1].cuda(),  # Attention masks [batch, seq_len]
        'label': batch[2].cuda(),          # Labels [batch]
        'category': batch[3].cuda(),       # Domain indices [batch]
        'image': batch[4].cuda(),          # MAE images [batch, 3, 224, 224]
        'clip_image': batch[5].cuda(),     # CLIP images [batch, 3, 224, 224]
        'clip_text': batch[6].cuda()       # CLIP text tokens [batch, 77]
    }
    return batch_data
```

--------------------------------

### Token Attention Mechanism with PyTorch

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Implements a self-attention mechanism for computing weighted feature representations of image tokens. It uses a sequential model to generate attention scores and then computes a weighted sum of the input features.

```python
import torch
import torch.nn as nn

class TokenAttention(torch.nn.Module):
    """Self-attention for computing weighted feature representations."""

    def __init__(self, input_shape):
        super(TokenAttention, self).__init__()
        self.attention_layer = nn.Sequential(
            torch.nn.Linear(input_shape, input_shape),
            nn.SiLU(),
            torch.nn.Linear(input_shape, 1),
        )

    def forward(self, inputs):
        # inputs: [batch, seq_len, dim]
        scores = self.attention_layer(inputs).view(-1, inputs.size(1))
        scores = scores.unsqueeze(1)
        outputs = torch.matmul(scores, inputs).squeeze(1)

        return outputs, scores  # [batch, dim], [batch, 1, seq_len]

# Example usage
image_attention = TokenAttention(768)
image_features = torch.randn(32, 197, 768)
image_output, scores = image_attention(image_features)  # [32, 768]
```

--------------------------------

### Masked Attention Mechanism with PyTorch

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Implements an attention mechanism for text sequences that supports padding masks. It calculates attention scores, applies the mask to ignore padded tokens, and computes a weighted sum of the input features.

```python
import torch
import torch.nn as nn

class MaskAttention(torch.nn.Module):
    """Attention with padding mask support for text sequences."""

    def __init__(self, input_dim):
        super(MaskAttention, self).__init__()
        self.Line = torch.nn.Linear(input_dim, 1)

    def forward(self, input, mask):
        # input: [batch, seq_len, dim]
        # mask: [batch, seq_len]
        score = self.Line(input).view(-1, input.size(1))

        if mask is not None:
            score = score.masked_fill(mask == 0, float("-inf"))

        score = torch.softmax(score, dim=-1).unsqueeze(1)
        output = torch.matmul(score, input).squeeze(1)

        return output  # [batch, dim]

# Example usage
text_attention = MaskAttention(768)
text_features = torch.randn(32, 197, 768)
masks = torch.ones(32, 197)
text_output = text_attention(text_features, masks)  # [32, 768]
```

--------------------------------

### Tokenize Chinese Text with BERT

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Converts raw text strings into token IDs and attention masks suitable for BERT-based models. It handles padding and truncation to ensure uniform input dimensions.

```python
from transformers import BertTokenizer
import torch

def word2input(texts, vocab_file, max_len):
    tokenizer = BertTokenizer(vocab_file=vocab_file)
    token_ids = []
    for text in texts:
        token_ids.append(
            tokenizer.encode(
                text,
                max_length=max_len,
                add_special_tokens=True,
                padding='max_length',
                truncation=True
            )
        )
    token_ids = torch.tensor(token_ids)
    masks = torch.zeros(token_ids.size())
    for i, token in enumerate(token_ids):
        masks[i] = (token != 0)
    return token_ids, masks
```

--------------------------------

### Compute Veracity-Specific Metrics

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Extends standard metrics to compute precision, recall, and F1-score specifically for fake and real news categories using a 0.5 probability threshold.

```python
def metricsTrueFalse(y_true, y_pred, category, category_dict):
    result = metrics(y_true, y_pred, category, category_dict)
    fake_TP, fake_FP, fake_FN = 0, 0, 0
    real_TP, real_FP, real_FN = 0, 0, 0
    y_pred_binary = [1 if p >= 0.5 else 0 for p in y_pred]

    for idx in range(len(y_pred_binary)):
        if y_true[idx] == 1:
            if y_pred_binary[idx] == 1:
                fake_TP += 1
            else:
                fake_FN += 1
                real_FP += 1
        else:
            if y_pred_binary[idx] == 0:
                real_TP += 1
            else:
                real_FN += 1
                fake_FP += 1

    result['fake'] = {
        'precision': fake_TP / max(1, fake_TP + fake_FP),
        'recall': fake_TP / max(1, fake_TP + fake_FN),
        'F1': 2 * (fake_TP / max(1, fake_TP + fake_FP)) * (fake_TP / max(1, fake_TP + fake_FN)) /
              max(1, (fake_TP / max(1, fake_TP + fake_FP)) + (fake_TP / max(1, fake_TP + fake_FN)))
    }
    result['real'] = {
        'precision': real_TP / max(1, real_TP + real_FP),
        'recall': real_TP / max(1, real_TP + real_FN),
        'F1': 2 * (real_TP / max(1, real_TP + real_FP)) * (real_TP / max(1, real_TP + real_FN)) /
              max(1, (real_TP / max(1, real_TP + real_FP)) + (real_TP / max(1, real_TP + real_FN)))
    }
    return result
```

--------------------------------

### MLP Classifier and Fusion Module in PyTorch

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Defines two PyTorch nn.Module classes: MLP for binary classification and MLP_fusion for fusing multi-modal features. Both modules utilize linear layers, batch normalization, GELU activation, and dropout. The MLP class outputs a single logit for classification, while MLP_fusion maps input features to a specified output dimension.

```python
import torch
import torch.nn as nn

class MLP(torch.nn.Module):
    """Multi-layer perceptron for binary classification."""

    def __init__(self, input_dim, embed_dims, dropout):
        """
        Args:
            input_dim: Input feature dimension (e.g., 320)
            embed_dims: List of hidden layer dimensions (e.g., [384])
            dropout: Dropout rate
        """
        super(MLP, self).__init__()
        layers = []

        for embed_dim in embed_dims:
            layers.append(torch.nn.Linear(input_dim, embed_dim))
            layers.append(torch.nn.BatchNorm1d(embed_dim))
            layers.append(torch.nn.GELU())
            layers.append(torch.nn.Dropout(p=dropout))
            input_dim = embed_dim

        layers.append(torch.nn.Linear(input_dim, 1))
        self.mlp = torch.nn.Sequential(*layers)

    def forward(self, x):
        return self.mlp(x)

class MLP_fusion(torch.nn.Module):
    """MLP for fusing multi-modal features."""

    def __init__(self, input_dim, out_dim, embed_dims, dropout):
        super(MLP_fusion, self).__init__()
        layers = []

        for embed_dim in embed_dims:
            layers.append(torch.nn.Linear(input_dim, embed_dim))
            layers.append(torch.nn.BatchNorm1d(embed_dim))
            layers.append(torch.nn.GELU())
            layers.append(torch.nn.Dropout(p=dropout))
            input_dim = embed_dim

        layers.append(torch.nn.Linear(input_dim, out_dim))
        self.mlp = torch.nn.Sequential(*layers)

    def forward(self, x):
        return self.mlp(x)

# Example usage
classifier = MLP(input_dim=320, embed_dims=[384], dropout=0.2)
fusion = MLP_fusion(input_dim=960, out_dim=320, embed_dims=[348], dropout=0.1)

features = torch.randn(32, 320)
logits = classifier(features)  # [32, 1]
probs = torch.sigmoid(logits)  # [32, 1]

```

--------------------------------

### Compute Classification Metrics for News Detection

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Calculates overall and domain-specific classification metrics including AUC, F1-score, recall, precision, and accuracy. It requires ground truth labels, predicted probabilities, and a domain category mapping.

```python
import numpy as np
from sklearn.metrics import recall_score, precision_score, f1_score, accuracy_score, roc_auc_score

def metrics(y_true, y_pred, category, category_dict):
    res_by_category = {}
    metrics_by_category = {}
    reverse_category_dict = {v: k for k, v in category_dict.items()}

    for k in category_dict.keys():
        res_by_category[k] = {"y_true": [], "y_pred": []}

    for i, c in enumerate(category):
        c = reverse_category_dict[c]
        res_by_category[c]['y_true'].append(y_true[i])
        res_by_category[c]['y_pred'].append(y_pred[i])

    try:
        metrics_by_category['auc'] = roc_auc_score(y_true, y_pred, average='macro')
    except ValueError:
        pass

    y_pred_binary = np.around(np.array(y_pred)).astype(int)
    metrics_by_category['metric'] = f1_score(y_true, y_pred_binary, average='macro')
    metrics_by_category['recall'] = recall_score(y_true, y_pred_binary, average='macro')
    metrics_by_category['precision'] = precision_score(y_true, y_pred_binary, average='macro')
    metrics_by_category['acc'] = accuracy_score(y_true, y_pred_binary)

    for c, res in res_by_category.items():
        y_pred_c = np.around(np.array(res['y_pred'])).astype(int)
        metrics_by_category[c] = {
            'precision': precision_score(res['y_true'], y_pred_c, average='macro'),
            'recall': recall_score(res['y_true'], y_pred_c, average='macro'),
            'fscore': f1_score(res['y_true'], y_pred_c, average='macro'),
            'acc': accuracy_score(res['y_true'], y_pred_c)
        }
    return metrics_by_category
```

--------------------------------

### CNN Feature Extractor with PyTorch

Source: https://context7.com/yutchina/mmdfnd/llms.txt

Applies multiple 1D convolutions with varying kernel sizes to extract multi-scale features from token sequences. It takes input features and a dictionary mapping kernel sizes to filter counts, returning concatenated pooled features.

```python
import torch
import torch.nn as nn

class cnn_extractor(torch.nn.Module):
    def __init__(self, input_size, feature_kernel):
        """
        Multi-scale CNN feature extractor.

        Args:
            input_size: Input feature dimension (e.g., 768)
            feature_kernel: Dict mapping kernel_size -> num_filters
                           e.g., {1: 64, 2: 64, 3: 64, 5: 64, 10: 64}
        """
        super(cnn_extractor, self).__init__()
        self.convs = torch.nn.ModuleList([
            torch.nn.Conv1d(input_size, feature_num, kernel)
            for kernel, feature_num in feature_kernel.items()
        ])

    def forward(self, input_data):
        # input_data: [batch, seq_len, features]
        input_data = input_data.permute(0, 2, 1)  # [batch, features, seq_len]

        feature = [conv(input_data) for conv in self.convs]
        feature = [torch.max_pool1d(f, f.shape[-1]) for f in feature]
        feature = torch.cat(feature, dim=1)
        feature = feature.view([-1, feature.shape[1]])

        return feature  # [batch, 320] (64*5 filters)

# Example usage
feature_kernel = {1: 64, 2: 64, 3: 64, 5: 64, 10: 64}
extractor = cnn_extractor(input_size=768, feature_kernel=feature_kernel)
text_features = torch.randn(32, 197, 768)  # [batch, seq_len, dim]
output = extractor(text_features)  # [32, 320]
```
Tokens:
Raw