QMSum

https://github.com/yale-lily/qmsum
Admin
QMSum is a human-annotated benchmark dataset for query-based multi-domain meeting summarization,...
Tokens:21,392
Snippets:380
Trust Score:8.6
License:MIT
Update:6 months ago
Show doc for...
Tokens:
Raw
### QMSum JSON Data Structure

Source: https://github.com/yale-lily/qmsum/blob/main/README.md

Example of the JSON schema used for QMSum dataset files, including topic lists, query lists, and meeting transcripts.

```json
{
    "topic_list": [
        {
            "topic": "Introduction of petitions and prioritization of governmental matters",
            "relevant_text_span": [["0","19"]]
        },
        {
            "topic": "Financial assistance for vulnerable Canadians during the pandemic and beyond",
            "relevant_text_span": [["21","57"], ["113","119"], ["191","217"]]
        },
        ...
    ],
    "general_query_list": [
        {
            "query": "Summarize the whole meeting.",
            "answer": "The meeting of the standing committee took place to discuss matters pertinent to the Coronavirus pandemic. The main issue at stake was to ..."
        },
        ...
    ],
    "specific_query_list": [
        {
            "query": "Summarize the discussion about introduction of petitions and prioritization of government matters.",
            "answer": "The Chair brought the meeting to order, announcing that the purpose of the meeting was to discuss COVID-19 's impact on Canada. Five petitions were presented ...",
            "relevant_text_span": [["0","19"]]
        },
	{
            "query": "What did Paul-Hus think about the introduction of petitions and prioritization of government matters?",
            "answer": "Mr. Paul-Hus thought that the government should not take firearms away from law-abiding Canadian citizens. He inquired into ...",
            "relevant_text_span": [["9","18"]]
        },
        ...
    ],
    "meeting_transcripts": [
        {
            "speaker": "The Chair (Hon. Anthony Rota (NipissingTimiskaming, Lib.))",
            "content": "I call the meeting to order.  Welcome to the third meeting of the House of Commons Special Committee on the COVID-19 Pandemic ..."
        },
        {
            "speaker": "Mr. Garnett Genuis (Sherwood ParkFort Saskatchewan, CPC)",
            "content": "Mr. Chair, I'm pleased to be presenting two petitions today. The first petition is with respect to government Bill C-7 ..."
        },
        ...
	{
            "speaker": "Hon. Seamus O'Regan",
            "content": "Mr. Chair, we have been working with our provincial partners. We have been working with businesses of all sizes in the oil and gas industry ...."
        },
        {
            "speaker": "The Chair",
            "content": "That's all the time we have for questions today. I want to thank all the members for taking part. The committee stands adjourned until tomorrow at noon.  The committee stands adjourned until tomorrow at noon. Thank you."
        }
    ]
}
```

--------------------------------

### Load QMSum Dataset

Source: https://github.com/yale-lily/qmsum/blob/main/data_process.ipynb

Reads a JSONL file from a specified path and stores meeting data in a list.

```python
import json

# read the dataset
# please enter the path of your data
split = 'train'
data_path = '/path/to/data/ALL/jsonl/' + split + '.jsonl'
data = []
with open(data_path) as f:
    for line in f:
        data.append(json.loads(line))
n_meetings = len(data)
print('Total {} meetings in the {} set.'.format(n_meetings, split))
```

--------------------------------

### Load and Analyze Domain-Specific Data

Source: https://context7.com/yale-lily/qmsum/llms.txt

Loads QMSum data splits by domain and calculates summary statistics for meetings and queries.

```python
def load_domain_data(domain):
    """
    Load all splits for a specific domain.

    Args:
        domain: 'Academic', 'Product', or 'Committee'

    Returns:
        Dictionary with train/val/test splits
    """
    return {
        'train': load_qmsum_data('train', domain),
        'val': load_qmsum_data('val', domain),
        'test': load_qmsum_data('test', domain)
    }

def get_domain_statistics(domain_data):
    """Calculate statistics for domain data."""
    stats = {}
    for split, meetings in domain_data.items():
        n_meetings = len(meetings)
        n_general = sum(len(m['general_query_list']) for m in meetings)
        n_specific = sum(len(m['specific_query_list']) for m in meetings)
        avg_turns = sum(len(m['meeting_transcripts']) for m in meetings) / n_meetings

        stats[split] = {
            'meetings': n_meetings,
            'general_queries': n_general,
            'specific_queries': n_specific,
            'avg_transcript_turns': round(avg_turns, 1)
        }
    return stats

# Load and analyze each domain
for domain in ['Academic', 'Product', 'Committee']:
    data = load_domain_data(domain)
    stats = get_domain_statistics(data)
    print(f"\n{domain} Domain:")
    for split, s in stats.items():
        print(f"  {split}: {s['meetings']} meetings, "
              f"{s['general_queries'] + s['specific_queries']} queries")
```

--------------------------------

### Load QMSum Data from JSONL

Source: https://context7.com/yale-lily/qmsum/llms.txt

Python function to load QMSum data from JSONL format for a specified split and domain. Requires the 'json' library.

```python
import json

# Load data from JSONL format
def load_qmsum_data(split='train', domain='ALL'):
    """
    Load QMSum data for specified split and domain.

    Args:
        split: 'train', 'val', or 'test'
        domain: 'ALL', 'Academic', 'Product', or 'Committee'

    Returns:
        List of meeting dictionaries
    """
    data_path = f'data/{domain}/jsonl/{split}.jsonl'
    data = []
    with open(data_path) as f:
        for line in f:
            data.append(json.loads(line))
    return data

# Load training data
train_data = load_qmsum_data('train', 'ALL')
print(f'Total {len(train_data)} meetings in training set')
# Output: Total 162 meetings in the train set.

# Access meeting components
meeting = train_data[0]
print(f"Topics: {len(meeting['topic_list'])}")
print(f"General queries: {len(meeting['general_query_list'])}")
print(f"Specific queries: {len(meeting['specific_query_list'])}")
print(f"Transcript turns: {len(meeting['meeting_transcripts'])}")
```

--------------------------------

### Process Meeting Transcripts for BART Input

Source: https://github.com/yale-lily/qmsum/blob/main/data_process.ipynb

Iterates through meeting data to construct source and target pairs for BART model training. It tokenizes and cleans the text, preparing it for summarization tasks. This script is designed for the QMSum dataset.

```python
bart_data_gold = []
for i in range(len(data)):
    # get meeting content
    entire_src = []
    for k in range(len(data[i]['meeting_transcripts'])):
        cur_turn = data[i]['meeting_transcripts'][k]['speaker'].lower() + ': '
        cur_turn = cur_turn + tokenize(data[i]['meeting_transcripts'][k]['content'])
        entire_src.append(cur_turn)
    entire_src = ' '.join(entire_src)
    for j in range(len(data[i]['general_query_list'])):
        cur = {}
        query = tokenize(data[i]['general_query_list'][j]['query'])
        cur['src'] = clean_data('<s> ' + query + ' </s> ' + entire_src + ' </s>')
        target = tokenize(data[i]['general_query_list'][j]['answer'])
        cur['tgt'] = target
        bart_data_gold.append(cur)
    for j in range(len(data[i]['specific_query_list'])):
        cur = {}
        query = tokenize(data[i]['specific_query_list'][j]['query'])
        src = []
        # get the content in the gold span for each query
        for span in data[i]['specific_query_list'][j]['relevant_text_span']:
            assert len(span) == 2
            st, ed = int(span[0]), int(span[1])
            for k in range(st, ed + 1):
                cur_turn = data[i]['meeting_transcripts'][k]['speaker'].lower() + ': '
                cur_turn = cur_turn + tokenize(data[i]['meeting_transcripts'][k]['content'])
                src.append(cur_turn)
        src = ' '.join(src)
        cur['src'] = clean_data('<s> ' + query + ' </s> ' + src + ' </s>')
        target = tokenize(data[i]['specific_query_list'][j]['answer'])
        cur['tgt'] = target
        bart_data_gold.append(cur)
        
print('Total {} query-summary pairs in the {} set'.format(len(bart_data_gold), split))
print(bart_data_gold[2])
with open('/path/to/data/bart_' + split + '._gold.jsonl', 'w') as f:
    for i in range(len(bart_data_gold)):
        print(json.dumps(bart_data_gold[i]), file=f)
```

--------------------------------

### Prepare BART Training Data

Source: https://github.com/yale-lily/qmsum/blob/main/data_process.ipynb

Iterates through meeting data to extract and format query-summary pairs for BART model training. Appends processed data to 'bart_data' list and writes to a JSONL file.

```python
bart_data = []
for i in range(len(data)):
    # get meeting content
    src = []
    for k in range(len(data[i]['meeting_transcripts'])):
        cur_turn = data[i]['meeting_transcripts'][k]['speaker'].lower() + ': '
        cur_turn = cur_turn + tokenize(data[i]['meeting_transcripts'][k]['content'])
        src.append(cur_turn)
    src = ' '.join(src)
    for j in range(len(data[i]['general_query_list'])):
        cur = {}
        query = tokenize(data[i]['general_query_list'][j]['query'])
        cur['src'] = clean_data('<s> ' + query + ' </s> ' + src + ' </s>')
        target = tokenize(data[i]['general_query_list'][j]['answer'])
        cur['tgt'] = target
        bart_data.append(cur)
    for j in range(len(data[i]['specific_query_list'])):
        cur = {}
        query = tokenize(data[i]['specific_query_list'][j]['query'])
        cur['src'] = clean_data('<s> ' + query + ' </s> ' + src + ' </s>')
        target = tokenize(data[i]['specific_query_list'][j]['answer'])
        cur['tgt'] = target
        bart_data.append(cur)
        
print('Total {} query-summary pairs in the {} set'.format(len(bart_data), split))
print(bart_data[2])
with open('/path/to/data/bart_' + split + '.jsonl', 'w') as f:
    for i in range(len(bart_data)):
        print(json.dumps(bart_data[i]), file=f)
```

--------------------------------

### Process QMSum Data for BART Model

Source: https://context7.com/yale-lily/qmsum/llms.txt

Converts QMSum data into source-target pairs for BART summarization. It combines query with meeting content, optionally using only relevant text spans for specific queries. Use this to prepare training data for sequence-to-sequence summarization tasks.

```python
import json

def process_for_bart(data, use_gold_spans=False):
    """
    Process QMSum data for BART summarization model.

    Args:
        data: List of meeting dictionaries
        use_gold_spans: If True, use only relevant text spans for specific queries

    Returns:
        List of {'src': source_text, 'tgt': target_summary} dictionaries
    """
    bart_data = []

    for meeting in data:
        # Build full meeting content
        full_src = []
        for turn in meeting['meeting_transcripts']:
            speaker = turn['speaker'].lower()
            content = tokenize(turn['content'])
            full_src.append(f"{speaker}: {content}")
        full_src_text = ' '.join(full_src)

        # Process general queries (use full meeting)
        for query_item in meeting['general_query_list']:
            query = tokenize(query_item['query'])
            target = tokenize(query_item['answer'])
            src = clean_data(f'<s> {query} </s> {full_src_text} </s>')
            bart_data.append({'src': src, 'tgt': target})

        # Process specific queries
        for query_item in meeting['specific_query_list']:
            query = tokenize(query_item['query'])
            target = tokenize(query_item['answer'])

            if use_gold_spans:
                # Extract only relevant spans
                src_parts = []
                for span in query_item['relevant_text_span']:
                    start, end = int(span[0]), int(span[1])
                    for idx in range(start, end + 1):
                        turn = meeting['meeting_transcripts'][idx]
                        speaker = turn['speaker'].lower()
                        content = tokenize(turn['content'])
                        src_parts.append(f"{speaker}: {content}")
                src_text = ' '.join(src_parts)
            else:
                src_text = full_src_text

            src = clean_data(f'<s> {query} </s> {src_text} </s>')
            bart_data.append({'src': src, 'tgt': target})

    return bart_data

# Process training data
train_data = load_qmsum_data('train', 'ALL')
bart_train = process_for_bart(train_data, use_gold_spans=False)
bart_train_gold = process_for_bart(train_data, use_gold_spans=True)

print(f'Total {len(bart_train)} query-summary pairs')
# Output: Total 1257 query-summary pairs in the train set

# Save processed data
with open('bart_train.jsonl', 'w') as f:
    for item in bart_train:
        print(json.dumps(item), file=f)
```

--------------------------------

### Evaluate Summaries with ROUGE

Source: https://context7.com/yale-lily/qmsum/llms.txt

Calculates ROUGE-1, ROUGE-2, and ROUGE-L F1 scores for generated summaries against references.

```python
from rouge_score import rouge_scorer

def evaluate_summaries(predictions, references):
    """
    Calculate ROUGE scores for generated summaries.

    Args:
        predictions: List of generated summary strings
        references: List of reference summary strings

    Returns:
        Dictionary with ROUGE-1, ROUGE-2, ROUGE-L F1 scores
    """
    scorer = rouge_scorer.RougeScorer(
        ['rouge1', 'rouge2', 'rougeL'],
        use_stemmer=True
    )

    scores = {'rouge1': [], 'rouge2': [], 'rougeL': []}

    for pred, ref in zip(predictions, references):
        score = scorer.score(ref, pred)
        for key in scores:
            scores[key].append(score[key].fmeasure)

    return {
        'rouge1': round(sum(scores['rouge1']) / len(scores['rouge1']) * 100, 2),
        'rouge2': round(sum(scores['rouge2']) / len(scores['rouge2']) * 100, 2),
        'rougeL': round(sum(scores['rougeL']) / len(scores['rougeL']) * 100, 2)
    }

# Example evaluation
preds, refs = load_model_outputs()
results = evaluate_summaries(preds, refs)
print(f"ROUGE-1: {results['rouge1']}")
print(f"ROUGE-2: {results['rouge2']}")
print(f"ROUGE-L: {results['rougeL']}")
# Expected HMNet output: 36.51/11.41/31.60
```

--------------------------------

### Load Extracted Spans and Model Outputs

Source: https://context7.com/yale-lily/qmsum/llms.txt

Utility functions to load pre-extracted spans from the Locator model and model prediction files.

```python
def load_extracted_spans(split='train'):
    """
    Load spans extracted by the Locator model.

    Args:
        split: 'train', 'val', or 'test'

    Returns:
        List of extracted span strings
    """
    spans = []
    with open(f'extracted_span/{split}.txt', 'r') as f:
        for line in f:
            spans.append(line.strip())
    return spans

def load_model_outputs():
    """
    Load HMNet model predictions and references.

    Returns:
        Tuple of (predictions, references) lists
    """
    with open('model_output/preds.txt', 'r') as f:
        preds = [line.strip() for line in f]
    with open('model_output/refs.txt', 'r') as f:
        refs = [line.strip() for line in f]
    return preds, refs

# Load extracted spans for training
train_spans = load_extracted_spans('train')
print(f"Loaded {len(train_spans)} extracted spans")

# Load model outputs for evaluation
preds, refs = load_model_outputs()
print(f"Loaded {len(preds)} predictions and {len(refs)} references")
# HMNet ROUGE scores: R-1/R-2/R-L = 36.51/11.41/31.60
```

--------------------------------

### Process Data for BART

Source: https://github.com/yale-lily/qmsum/blob/main/data_process.ipynb

This snippet is used for processing data specifically for the BART model. Ensure all necessary libraries are imported before use.

```python
# process data for BART

```

--------------------------------

### Tokenize Sentences

Source: https://github.com/yale-lily/qmsum/blob/main/data_process.ipynb

Converts a sentence to lowercase and tokenizes it using NLTK word_tokenize.

```python
from nltk import word_tokenize
# tokneize a sent
def tokenize(sent):
    tokens = ' '.join(word_tokenize(sent.lower()))
    return tokens
```
Tokens:
Raw