### Install selective-context and spaCy models

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Install the selective-context library and necessary spaCy language models for English and optionally Chinese.

```bash
pip install selective-context
python -m spacy download en_core_web_sm
```

```bash
python -m spacy download zh_core_web_sm
```

--------------------------------

### Install Selective Context and spaCy Models

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Install the library using pip and download necessary spaCy language models for English and Chinese support.

```bash
pip install selective-context
python -m spacy download en_core_web_sm
# For Chinese support:
python -m spacy download zh_core_web_sm
```

--------------------------------

### Get Current Working Directory

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

This magic command displays the current working directory.

```python
%pwd
```

--------------------------------

### Access Element in Dataset

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Accesses a specific element in a `Dataset` object by its index. This example retrieves the element at index 4.

```python
ds[4]
```

--------------------------------

### Main Function for Aggregating Results by Context Type

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Aggregates results from various tasks and data sources for a given context type across different mask ratios. It processes data, calculates averages, and concatenates results into a single DataFrame. Call this function to get aggregated performance metrics for a specific context strategy.

```python
def main(context_type):
    tasks = ['summarisation', 'qa', 'reconstruction', 'continue_conversation']
    data_sources = ['news', 'arxiv']
    mask_ratios = [0.2, 0.35, 0.5, 0.65, 0.8]
    
    all_results = []
    method = f"Original"
    for mask in mask_ratios:
        avg_results = read_results(tasks=tasks, data_sources=data_sources, mask_ratios=[mask], context_type=context_type)
        avg_results['Method'] = context_type
        avg_results['Ratio'] = mask
        avg_r = avg_results.to_frame().T.set_index(['Method', 'Ratio'])
        print(avg_r)
        all_results.append(avg_r)
    df = pd.concat(all_results)
    # print(df)
    # ratio_avg = df.mean()
    # ratio_avg.name
    # df.loc[(method, 'avg'), :] = ratio_avg
    # print(df, "^^^^^^^^^^")
    # all_results.append(df)
    
    # df = pd.concat(all_results, axis=0)
    print(df)
    return df
```

--------------------------------

### Download Datasets and Run Experiments

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Commands to download necessary datasets and execute the main script for reproducing paper experiments.

```bash
# Download datasets
wget https://github.com/liyucheng09/Selective_Context/releases/download/v0.1.0rc1/datasets_dumps.zip
unzip datasets_dumps.zip

# Run experiments
python main.py \
    datasets_dumps/arxiv \
    datasets_dumps/news \
    datasets_dumps/conversation \
    ./results \
    50 \
    gpt-3.5-turbo
```

--------------------------------

### Download and unzip datasets

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Download the datasets required for experiments and unzip them for use.

```bash
wget https://github.com/liyucheng09/Selective_Context/releases/download/v0.1.0rc1/datasets_dumps.zip
unzip datasets_dumps.zip
```

--------------------------------

### Run the streamlit app

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Launch the web interface for Selective Context using the provided streamlit command.

```bash
streamlit run app/app.py
```

--------------------------------

### Run Streamlit Web Interface

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Command to run the Streamlit application for interactive text compression.

```bash
# Run the Streamlit app
streamlit run app/app.py

# Or access the hosted demo at:
# https://huggingface.co/spaces/liyucheng/selective_context
```

--------------------------------

### Run experiments

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Execute the main script to run experiments, providing paths to datasets, output directory, number of articles, and Hugging Face model name.

```bash
python main.py datasets_dumps/arxiv datasets_dumps/news datasets_dump/conversation <output_path_to_save_results> <num_articles> <HF_model_name_or_path>
```

--------------------------------

### Select Reduction Level for Text Compression (Python)

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Demonstrates text compression using SelectiveContext with different reduction levels: 'sent' (sentence), 'phrase' (default), and 'token'. Phrase-level offers a balance between compression and readability. Requires 'selective_context' library.

```python
from selective_context import SelectiveContext

sc = SelectiveContext(model_type='gpt2', lang='en')

text = """
The transformer architecture has revolutionized natural language processing since its
introduction in 2017. It relies entirely on self-attention mechanisms to compute
representations of input and output sequences. Unlike recurrent neural networks,
transformers can process all positions in parallel during training. This parallelization
enables significantly faster training on modern GPU hardware. The architecture consists
of an encoder and decoder, each composed of multiple identical layers.
"""

# Sentence-level reduction (removes entire sentences)
sent_context, sent_removed = sc(text, reduce_ratio=0.5, reduce_level='sent')
print("Sentence-level compression:")
print(sent_context)
print(f"Removed sentences: {sent_removed}\n")

# Phrase-level reduction (removes noun phrases - default)
phrase_context, phrase_removed = sc(text, reduce_ratio=0.5, reduce_level='phrase')
print("Phrase-level compression:")
print(phrase_context)
print(f"Removed phrases: {phrase_removed}\n")

# Token-level reduction (removes individual tokens)
token_context, token_removed = sc(text, reduce_ratio=0.5, reduce_level='token')
print("Token-level compression:")
print(token_context)
print(f"Removed tokens: {token_removed}")
```

--------------------------------

### Compress prompt and context with custom reduce ratio

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Compress text using SelectiveContext, specifying a custom 'reduce_ratio' to control the compression level.

```python
context, reduced_content = sc(text, reduce_ratio = 0.5)
```

--------------------------------

### Download data dumps

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Download data dumps if there are issues accessing Huggingface Hub.

```bash
wget https://github.com/liyucheng09/Selective_Context/releases/download/v0.1.0rc1/data_dumps.zip
```

--------------------------------

### Initialize and Use SelectiveContext for Compression

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Initialize the SelectiveContext class with a specified model type and language. Then, pass the long text to the instance to compress it. The method returns the compressed context and the removed content.

```python
from selective_context import SelectiveContext

# Initialize the context compressor
sc = SelectiveContext(model_type='gpt2', lang='en')

# Sample long text to compress
long_text = """
Machine learning is a subset of artificial intelligence that enables systems to learn
and improve from experience without being explicitly programmed. It focuses on developing
computer programs that can access data and use it to learn for themselves. The process
begins with observations or data, such as examples, direct experience, or instruction.
It looks for patterns in data and makes better decisions in the future based on the
examples provided. The primary aim is to allow computers to learn automatically without
human intervention and adjust actions accordingly. Deep learning is a subset of machine
learning that uses neural networks with many layers. These neural networks attempt to
simulate the behavior of the human brain in processing data and creating patterns.
"""

# Compress with default settings (35% reduction at phrase level)
compressed_context, removed_content = sc(long_text)

print("Compressed Context:")
print(compressed_context)
print("\nRemoved Content:")
print(removed_content)
```

--------------------------------

### Create and Combine Lexical Units

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Demonstrates creating LexicalUnits for sentences and phrases, combining them, and adding units to the head.

```python
from selective_context import LexicalUnits

# Create lexical units manually
sentences = LexicalUnits(
    unit_type='sent',
    text=['First sentence.', 'Second sentence.', 'Third sentence.'],
    self_info=[2.5, 1.2, 3.8]  # Self-information scores
)

phrases = LexicalUnits(
    unit_type='phrase',
    text=['Natural language', 'processing', 'is', 'a field'],
    self_info=[3.2, 2.1, 0.5, 2.8]
)

# Combine lexical units of the same type
more_sentences = LexicalUnits(
    unit_type='sent',
    text=['Fourth sentence.'],
    self_info=[2.0]
)
combined = sentences + more_sentences
print(f"Combined sentences: {combined.text}")
# Output: ['First sentence.', 'Second sentence.', 'Third sentence.', 'Fourth sentence.']

# Add tokens to head or tail
extended = sentences.add_to_head('Intro: ', 5.0)
print(f"Extended text: {extended.text}")
# Output: ['Intro: ', 'First sentence.', 'Second sentence.', 'Third sentence.']
```

--------------------------------

### Process Article Prompts

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Iterates through articles, constructs a prompt from context, and appends it to each article object. Assumes articles have a 'units' attribute and 'context' is a list of (role, utterance) tuples.

```python
articles = []
for article in data.articles:
    prompt = ''
    for role, utterance in article.context[:-1]:
        prompt += f"{role}: {utterance}\n"
    prompt += 'gpt: '
    article.prompt = prompt
    articles.append(article)
```

--------------------------------

### Instantiate Parser Object

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Creates an instance of the parser class, specifying the directory for articles.

```python
manager = parser('/vol/research/lyc/llm_memorize/conversation')
```

--------------------------------

### Compress prompt and context

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Initialize SelectiveContext and use it to compress a given text. The 'context' variable will hold the compressed version.

```python
sc = SelectiveContext(model_type='gpt2', lang='en')
context, reduced_content = sc(text)
```

--------------------------------

### Process Academic Papers with ArxivContextManager (Python)

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Demonstrates batch processing of arXiv papers using ArxivContextManager. It handles self-information scoring and context compression with various masking strategies. Requires 'context_manager' library.

```python
from context_manager import ArxivContextManager

# Initialize manager for processing arXiv papers
manager = ArxivContextManager(
    path='./arxiv_data',       # Directory containing JSON article files
    mask_ratio=0.35,           # Percentage of content to mask
    keep_leading_word=True,    # Keep leading words when masking sentences
    num_lead_words=3,          # Number of leading words to keep
    num_articles=100,          # Maximum number of articles to process
    lang='en'                  # Language setting
)

# Generate contexts with self-information based masking at phrase level
contexts = manager.generate_context(
    mask_method='self-info',   # Options: 'self-info', 'Random', 'no'
    mask_level='phrase',       # Options: 'sent', 'phrase', 'token'
    num_articles=50            # Number of articles to process
)

# Access compressed contexts
for ctx in contexts[:3]:
    print(f"Article ID: {ctx.id}")
    print(f"Compressed context: {ctx.context[:200]}...")
    print(f"Masked content: {ctx.masked_sents[:3] if ctx.masked_sents else 'None'}")
    print("---")

# Save/restore from checkpoint for long-running processes
manager._check_point('Processing checkpoint')

# Load from checkpoint
restored_manager = ArxivContextManager.from_checkpoint(
    './arxiv_data/ArxivContextManager_sent.pkl',
    phrase_mask_token='',
    max_token_len=1200
)
```

--------------------------------

### Chinese Language Text Compression (Python)

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Shows how to compress Chinese text using SelectiveContext by initializing it with 'lang="zh"'. Requires 'selective_context' and a Chinese GPT-2 model.

```python
from selective_context import SelectiveContext

# Initialize for Chinese
sc_zh = SelectiveContext(model_type='gpt2', lang='zh')

chinese_text = """
人工智能是计算机科学的一个分支，它试图理解智能的本质，并生产出一种新的能以人类
智能相似的方式做出反应的智能机器。该领域的研究包括机器人、语言识别、图像识别、
自然语言处理和专家系统等。人工智能从诞生以来，理论和技术日益成熟，应用领域也
不断扩大，可以设想未来人工智能带来的科技产品将会是人类智慧的容器。
"""

# Compress Chinese text
compressed_zh, removed_zh = sc_zh(chinese_text, reduce_ratio=0.35)

print("Compressed Chinese text:")
print(compressed_zh)
print("\nRemoved content:")
print(removed_zh)
```

--------------------------------

### Import SelectiveContext

Source: https://github.com/liyucheng09/selective_context/blob/main/readme.md

Import the SelectiveContext class from the library to begin using its functionality.

```python
from selective_context import SelectiveContext
```

--------------------------------

### Initialize GPT-2 Tokenizer

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Loads the GPT-2 tokenizer from the transformers library. This is necessary for text tokenization tasks.

```python
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
```

--------------------------------

### Load Data from Pickle File

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Loads data from a specified pickle file. Ensure the file path is correct and the pickle module is imported.

```python
import pickle

with open('/vol/research/lyc/llm_memorize/news/NewsContextManager_sent.pkl', 'rb') as f:
    data = pickle.load(f)
```

--------------------------------

### Creating a List of Article Dictionaries

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Generate a list of dictionaries, where each dictionary contains the 'id' and 'chat' context for an article from the manager's articles.

```Python
covs = [{'id': article.entry_id, 'chat': article.context} for article in manager.articles]
```

--------------------------------

### Initialize and Generate Compressed Conversations

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Initializes the ConversationContextManager with specified parameters and generates compressed conversations using a self-information masking method.

```python
from context_manager import ConversationContextManager

# Initialize conversation manager
conv_manager = ConversationContextManager(
    path='./conversation_data',  # Directory with conversation_2k.json
    mask_ratio=0.35,
    num_articles=100,
    lang='en'
)

# Generate compressed conversations
compressed_convs = conv_manager.generate_context(
    mask_method='self-info',
    mask_level='phrase'
)
```

--------------------------------

### Programmatic Text Compression with SelectiveContext

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Provides a programmatic equivalent to the Streamlit interface for text compression, returning compressed text and removed phrases.

```python
# The Streamlit app provides:
# - Language selection (English/Chinese)
# - Compression ratio slider (0.2, 0.35, 0.5, 0.65, 0.8)
# - Reduction level selection (phrase, token, sent)
# - Text input area
# - Compressed output display
# - Filtered content display

# Example programmatic equivalent:
from selective_context import SelectiveContext

def compress_interactive(text, lang='en', ratio=0.5, level='phrase'):
    sc = SelectiveContext(model_type='gpt2', lang=lang)
    compressed, removed = sc(text, reduce_ratio=ratio, reduce_level=level)
    return {
        'compressed': compressed,
        'removed': removed,
        'reduction_percent': (1 - len(compressed) / len(text)) * 100
    }

result = compress_interactive(
    text="Your long text here...",
    lang='en',
    ratio=0.5,
    level='phrase'
)
print(f"Achieved {result['reduction_percent']:.1f}% reduction")
```

--------------------------------

### Initialize Parser Class

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Initializes the parser class with a specified path. This class is designed to manage and process conversational data.

```python
class parser(ConversationContextManager):
    def __init__(
        self,
        path : str,
        mask_ratio = 0.2, 
        keep_leading_word = True,
        num_lead_words = 3,
        ppl_threshold = None,
        tokenizer = None,
        compute_self_info = True,
        sent_mask_token = "<...some content omitted.>",
        phrase_mask_token = "",
        num_articles = 2000,
        lang = "en",
    ):
        self.path = path
        self.lang = lang
        self._prepare_phrase_tokenizer()
        self.num_articles = num_articles
        self.tokenizer = GPT2Tokenizer.from_pretrained("gpt2") if tokenizer is None else tokenizer

        self.keep_leading_word = keep_leading_word
        self.num_lead_words = num_lead_words
        self.ppl_threshold = ppl_threshold
        self.max_token_len = 1800
        self.sent_level_self_info = True
        self.mask_ratio = mask_ratio

        self.mask_token = sent_mask_token
        self.phrase_mask_token = phrase_mask_token

        # self.sent_tokenize_pattern = r"((?<!e.g)(?<!i.e)(?<!w.r.t)(?<=\.)\s)|(?<=\?\s)|(?<=!\s)"
        # self.sent_tokenize_pattern = r"(?<!e.g)(?<!i.e)(?<=\.\s)|(?<=\?\s)|(?<=!\s)"
        self.sent_tokenize_pattern = r"(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s"
        self.load_articles(path)
```

--------------------------------

### Generate Contexts with Masking - Python

Source: https://context7.com/liyucheng09/selective_context/llms.txt

This snippet demonstrates how to configure and generate contexts using ArxivContextManager with different masking ratios and types. Ensure the necessary classes (ArxivContextManager, NewsContextManager, etc.) are imported.

```python
from context_manager import ArxivContextManager, NewsContextManager
from qa_manager import Summarisation, QA, OriginalContextReconsutrction

# Configuration
mask_types = ['no', 'self-info', 'Random']
mask_levels = ['phrase']
task_types = ['reconstruction', 'summarisation', 'qa']
mask_ratios = [0.2, 0.35, 0.5, 0.65, 0.8]

# Initialize dataset managers
arxiv_manager = ArxivContextManager('./datasets_dumps/arxiv', num_articles=100)
news_manager = NewsContextManager('./datasets_dumps/news', num_articles=100)

# Generate contexts with different configurations
for mask_ratio in mask_ratios:
    arxiv_manager.mask_ratio = mask_ratio

    for mask_type in mask_types:
        if mask_type == 'no':
            contexts = arxiv_manager.generate_context('no', num_articles=50)
        else:
            contexts = arxiv_manager.generate_context(
                mask_type,
                mask_level='phrase',
                num_articles=50
            )

        print(f"Generated {len(contexts)} contexts with {mask_type} masking at {mask_ratio} ratio")
```

--------------------------------

### Push Dataset to Hub

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Use this command to push a dataset to the Hugging Face Hub. Ensure the dataset is in Arrow format.

```python
ds.push_to_hub('liyucheng/sharegpt-500')
```

--------------------------------

### Add Multiple Columns by Chaining Table.AddColumn

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Use this method to add new columns sequentially by calling Table.AddColumn multiple times. Ensure each step references the output of the previous one.

```M
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUzI1V...", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t]),
    #
```

```M
Added Custom Columns" = Table.AddColumn(Source, "Custom Column 1", each "Value 1"),
    #
```

```M
Added Custom Columns1" = Table.AddColumn(#"Added Custom Columns", "Custom Column 2", each "Value 2"),
    #
```

```M
Added Custom Columns2" = Table.AddColumn(#"Added Custom Columns1", "Custom Column 3", each "Value 3")
in
    #
```

```M
Added Custom Columns2"
```

--------------------------------

### Plot Data Source Performance Metrics

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Visualizes BLEU, ROUGE1, and BERTScore metrics across different context reduction ratios for various data sources. Requires matplotlib and pandas.

```python
import matplotlib.pyplot as plt

df = data_wise('self-info-phrase')
df = df.reset_index()

fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 4), dpi=120)

markers = {
    'arxiv': '+',
    'BBC': 's',
    'ShareGPT': 'v',
}

colors = {
    'arxiv': 'salmon',
    'BBC': 'y',
    'ShareGPT': 'violet',
}

for data_source, group in df.groupby('Data'):
    if data_source == 'news':
        data_source = 'BBC'
    if data_source == 'conversation':
        data_source = 'ShareGPT'
    group.plot(y='bleu', x='Ratio', ax=axes[0], marker=markers[data_source], label = data_source, color=colors[data_source])
    group.plot(y='rouge1', x='Ratio', ax=axes[1], marker=markers[data_source], label = data_source, color=colors[data_source])
    group.plot(y='bertscore_f1', x='Ratio', ax=axes[2], marker=markers[data_source], label = data_source, color=colors[data_source])

axes[0].set_title('BLEU')
axes[0].set_xlabel('')
axes[0].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])

axes[1].set_title('ROUGE1')
axes[1].set_xlabel('')
axes[1].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])

axes[2].set_title('BERTScore')
axes[2].set_xlabel('')
axes[2].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])
fig.text(0.52, -0.03, 'Context reduction ratio', ha='center', fontsize=14)
plt.tight_layout()
```

--------------------------------

### Run Main Analysis with 'no2-phrase' Context

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Executes the main analysis function with the 'no2-phrase' context type. This likely represents a baseline or a specific condition for comparison.

```python
main('no2-phrase')
```

--------------------------------

### Process News Articles with NewsContextManager (Python)

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Illustrates using NewsContextManager to process BBC news articles from Hugging Face Hub. It generates compressed contexts with self-information masking. Requires 'context_manager' library.

```python
from context_manager import NewsContextManager

# Initialize news context manager
news_manager = NewsContextManager(
    path='./news_data',        # Path for cache/checkpoints
    mask_ratio=0.5,            # 50% content reduction
    num_articles=200,          # Process up to 200 articles
    lang='en'
)

# Generate compressed news contexts
news_contexts = news_manager.generate_context(
    mask_method='self-info',
    mask_level='phrase',
    num_articles=100
)

# Process results
for article in news_contexts[:5]:
    original_length = len(article.text)
    compressed_length = len(article.context)
    reduction = (1 - compressed_length / original_length) * 100

    print(f"Article: {article.entry_id}")
    print(f"Original: {original_length} chars -> Compressed: {compressed_length} chars")
    print(f"Reduction: {reduction:.1f}%")
    print("---")
```

--------------------------------

### Import GPT2Tokenizer

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the GPT2Tokenizer from the transformers library, likely for text processing.

```python
from transformers import GPT2Tokenizer
```

--------------------------------

### Bitsandbytes GPU Support Warning

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

A UserWarning from bitsandbytes indicating that the library was compiled without GPU support. This means GPU-accelerated features are unavailable.

```text
/user/HS502/yl02706/.conda/envs/lyc/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "

```

--------------------------------

### Generate and Compare DataFrames from Main Function

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Executes the main aggregation function twice with different context types ('Random-phrase' and 'self-info-phrase') and assigns the results to df and df2 respectively. This is used for comparing the performance of different context strategies.

```python
df = main('Random-phrase')
df2 = main('self-info-phrase')
```

--------------------------------

### Process and Aggregate Results by Method and Task

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Processes and aggregates results from multiple runs, organizing them by method (e.g., 'SC-ratio') and task. It calculates average metrics for each combination and appends an overall average row. Requires 'pandas'.

```python
def main(context_type):
    tasks = ['summarisation', 'qa', 'conversation']
    data_sources = ['news', 'arxiv',]
    mask_ratios = [0.2, 0.35, 0.5, 0.65, 0.8]
    
    all_results = []
    for ratio in mask_ratios:
        ratio_results = []
        method = f"SC-{ratio}"
        for task in tasks:
            if task == 'conversation':
                avg_results = read_results(tasks=['continue_conversation'], data_sources=['conversation'], mask_ratios=[ratio], context_type=context_type)
            else:
                avg_results = read_results(tasks=[task], data_sources=data_sources, mask_ratios=[ratio], context_type=context_type)
            avg_results['Method'] = method
            avg_results['Task'] = task
            avg_r = avg_results.to_frame().T.set_index(['Method', 'Task'])
            print(ratio, '----', task)
            print(avg_r)
            ratio_results.append(avg_r)
        print(ratio, '*****')
        df = pd.concat(ratio_results)
        print(df)
        ratio_avg = df.mean()
        ratio_avg.name
        df.loc[(method, 'avg'), :] = ratio_avg
        print(df, "^^^^^^^^^")
        all_results.append(df)
    
    df = pd.concat(all_results, axis=0)
    print(df)
    return df

```

```python
df = main('self-info-phrase')
```

--------------------------------

### Print Article Prompt

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Prints the generated prompt for the third article (index 2). Useful for debugging prompt construction.

```python
print(data.articles[2].prompt)
```

--------------------------------

### Bitsandbytes CPU Compilation Warning

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

This output indicates that the bitsandbytes library was compiled without GPU support. Features like 8-bit optimizers and GPU quantization will not be available.

```text
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /user/HS502/yl02706/.conda/envs/lyc/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/user/HS502/yl02706/.conda/envs/lyc/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
CUDA SETUP: Loading binary /user/HS502/yl02706/.conda/envs/lyc/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so...

```

--------------------------------

### Run Main Analysis with 'Random-phrase' Context

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Executes the main analysis function with the 'Random-phrase' context type. This is used to evaluate the model's performance when the context is randomly selected.

```python
main('Random-phrase')
```

--------------------------------

### Change Directory Command

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Changes the current working directory to the parent directory.

```python
%cd ..
```

--------------------------------

### Plot Task-wise Performance Metrics

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Visualizes BLEU, ROUGE1, and BERTScore metrics across different context reduction ratios for various tasks. Requires matplotlib and pandas.

```python
import matplotlib.pyplot as plt

df = task_wise('self-info-phrase')
df = df.reset_index()

fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 4), dpi=120)

markers = {
    'Conversation': '+',
    'Reconstruction': 's',
    'Summarisation': 'v',
    'QA': '^',
}

colors = {
    'Conversation': 'salmon',
    'Reconstruction': 'y',
    'Summarisation': 'grey',
    'QA': 'violet',
}

for task_name, group in df.groupby('Task'):
    if task_name == 'continue_conversation':
        task_name = 'Conversation'
    if task_name == 'reconstruction':
        task_name = 'Reconstruction'
    if task_name == 'summarisation':
        task_name = 'Summarisation'
    if task_name == 'qa':
        task_name = 'QA'
    group.plot(y='bleu', x='Ratio', ax=axes[0], marker=markers[task_name], label = task_name, color=colors[task_name])
    group.plot(y='rouge1', x='Ratio', ax=axes[1], marker=markers[task_name], label = task_name, color=colors[task_name])
    group.plot(y='bertscore_f1', x='Ratio', ax=axes[2], marker=markers[task_name], label = task_name, color=colors[task_name])

axes[0].set_title('BLEU')
axes[0].set_xlabel('')
axes[0].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])

axes[1].set_title('ROUGE1')
axes[1].set_xlabel('')
axes[1].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])

axes[2].set_title('BERTScore')
axes[2].set_xlabel('')
axes[2].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])
fig.text(0.52, -0.03, 'Context reduction ratio', ha='center', fontsize=14)
plt.tight_layout()
```

--------------------------------

### Import Datasets Library in Python

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the `Dataset` class from the `datasets` library, commonly used for handling and processing large datasets.

```python
from datasets import Dataset
```

--------------------------------

### Run Main Analysis with 'self-info-phrase' Context

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Executes the main analysis function with the 'self-info-phrase' context type. This involves iterating through mask ratios and calling the read_results function to gather performance metrics.

```python
def main(context_type):
    tasks = ['summarisation', 'qa', 'continue_conversation']
    data_sources = ['news', 'arxiv']
    mask_ratios = [0.2, 0.35, 0.5, 0.65, 0.8]
    
    all_results = {}
    for ratio in mask_ratios:
        avg_results = read_results(tasks=tasks, data_sources=data_sources, mask_ratios=[ratio], context_type=context_type)
        print(avg_results)
        all_results[ratio] = avg_results
    
    df = pd.DataFrame.from_dict(all_results, orient='index')
    print(df)

```

```python
main('self-info-phrase')
```

--------------------------------

### Configure Reduction Ratio for Compression

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Adjust the `reduce_ratio` parameter when calling the SelectiveContext instance to control the percentage of content filtered out. Higher values lead to more aggressive compression. The output shows the number of phrases removed for different reduction ratios.

```python
from selective_context import SelectiveContext

sc = SelectiveContext(model_type='gpt2', lang='en')

text = """
Natural language processing is a field of computer science and linguistics concerned
with the interactions between computers and human language. It involves programming
computers to process and analyze large amounts of natural language data. The result
is a computer capable of understanding the contents of documents, including the
contextual nuances of the language within them. NLP can accurately extract information
and insights contained in the documents as well as categorize and organize the
documents themselves.
"""

# Light compression (20% removed)
light_compressed, light_removed = sc(text, reduce_ratio=0.2)
print(f"Light compression - Removed {len(light_removed)} phrases")

# Medium compression (50% removed) - recommended
medium_compressed, medium_removed = sc(text, reduce_ratio=0.5)
print(f"Medium compression - Removed {len(medium_removed)} phrases")

# Heavy compression (80% removed)
heavy_compressed, heavy_removed = sc(text, reduce_ratio=0.8)
print(f"Heavy compression - Removed {len(heavy_removed)} phrases")
```

--------------------------------

### Import ConversationContextManager

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the ConversationContextManager class from the context_manager module.

```python
from context_manager import ConversationContextManager
```

--------------------------------

### Load Pickled Data

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Loads data from a pickle file. Ensure the file path is correct and the file exists before execution.

```python
with open('/vol/research/lyc/llm_memorize/answer_summarisation_arxiv_0.35.pkl', 'rb') as f:
    d = pickle.load(f)
```

--------------------------------

### Process Text File from Command Line

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Defines a function to process a text file, compress its content, and save the output to another file, printing statistics.

```python
# From selective_context.py main function:

# Interactive mode - process text from stdin
# python selective_context.py

# Process a file and save output
from selective_context import SelectiveContext

def process_file(input_path, output_path, model_type='gpt2', lang='en', reduce_ratio=0.35):
    """Process a text file and save compressed output."""
    sc = SelectiveContext(model_type=model_type, lang=lang)

    with open(input_path, 'r', encoding='utf-8') as f:
        text = f.read()

    compressed, removed = sc(text, reduce_ratio=reduce_ratio)

    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(compressed)

    print(f"Original: {len(text)} chars")
    print(f"Compressed: {len(compressed)} chars")
    print(f"Removed {len(removed)} phrases")

    return compressed, removed

# Example usage
compressed, removed = process_file(
    input_path='./documents/long_article.txt',
    output_path='./documents/compressed_article.txt',
    reduce_ratio=0.5
)
```

--------------------------------

### Import JSON Library

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the JSON library for potential JSON manipulation.

```python
import json
```

--------------------------------

### Load Pickled Data

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Loads data from a pickle file. Ensure the file path is correct and the file exists.

```python
import pickle
with open('/vol/research/lyc/llm_memorize/news/NewsContextManager_sent.pkl', 'rb') as f:
    news = pickle.load(f)
```

--------------------------------

### Visualize Performance Metrics

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Generates plots for BLEU, ROUGE1, and BERTScore F1 across different context reduction ratios. It compares 'Selective Context' (df2) with 'Random' (df) methods. Ensure df and df2 are pre-processed with reset_index() before plotting.

```python
import matplotlib.pyplot as plt

df2 = df2.reset_index()
df = df.reset_index()

fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 4), dpi=120)
df2.plot(y='bleu', x='Ratio', ax=axes[0], marker='^', label = 'Selective Context')
df.plot(y='bleu', x='Ratio', ax=axes[0], marker='+', label = 'Random')
axes[0].set_title('BLEU')
axes[0].set_xlabel('')
axes[0].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])
# axes[0].set_xlabel('Filtered Ratio')

df2.plot(y='rouge1', x='Ratio', ax=axes[1], marker='^', label = 'Selective Context')
df.plot(y='rouge1', x='Ratio', ax=axes[1], marker='+', label = 'Random')
axes[1].set_title('ROUGE1')
axes[1].set_xlabel('')
axes[1].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])
# axes[1].set_ylim(0.2, 0.7)
# axes[1].set_xlabel('Filtered Ratio')

df2.plot(y='bertscore_f1', x='Ratio', ax=axes[2], marker='^', label = 'Selective Context')
df.plot(y='bertscore_f1', x='Ratio', ax=axes[2], marker='+', label = 'Random')
axes[2].set_title('BERTScore')
axes[2].set_xlabel('')
axes[2].set_xticks([0.2, 0.35, 0.5, 0.65, 0.8])
# axes[-1].set_xlabel('Filtered Ratio')
# axes[2].set_ylim(0.5, 1)
fig.text(0.52, -0.03, 'Context reduction ratio', ha='center', fontsize=14)
plt.tight_layout()
```

--------------------------------

### Read LLM Memorization Results

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Reads and processes pickled results from LLM memorization experiments. Calculates average BERTScore metrics and returns a Pandas DataFrame of aggregated results. Requires 'pickle', 'pandas', and 'numpy' libraries.

```python
from typing import List
import pickle
import pandas as pd
import numpy as np

def read_results(tasks: List[str], data_sources: List[str], mask_ratios: List[str], context_type,):
    
    all_ans_paths = []
    for task in tasks:
        if task != 'continue_conversation':
            for data_source in data_sources:
                for mask_ratio in mask_ratios:
                    if data_source == 'news' or data_source == 'conversation':
                        ans_path = f'/vol/research/lyc/llm_memorize/answer_{task}_{data_source}_{mask_ratio}.pkl'
                    elif data_source == 'arxiv':
                        ans_path = f'/vol/research/lyc/llm_memorize/{"arxiv_buggy/" if context_type == "Random-phrase" else ""}answer_{task}_{data_source}_{mask_ratio}.pkl'
                    all_ans_paths.append(ans_path)
        else:
            for mask_ratio in mask_ratios:
                ans_path = f'/vol/research/lyc/llm_memorize/answer_{task}_conversation_{mask_ratio}.pkl'
                all_ans_paths.append(ans_path)
    
    results = []
    for ans_path in all_ans_paths:
        print(ans_path)

        with open(ans_path, 'rb') as f:
            ans = pickle.load(f)
        
        r = ans.metrics[context_type]
        if 'precision' in r:
            r['bertscore_precision'] = np.mean(r['precision'])
            r['bertscore_recall'] = np.mean(r['recall'])
            r['bertscore_f1'] = np.mean(r['f1'])
        results.append(r)
    
    df = pd.DataFrame(results)
    df = df[['bleu', 'meteor', 'rouge1', 'rouge2', 'rougeL', 'bertscore_precision', 'bertscore_recall', 'bertscore_f1']]
    avg_results = df.mean()

    return avg_results
```

--------------------------------

### Accessing the First Article's Context

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Retrieve the first item from the 'covs' list, which contains the 'id' and 'chat' context of the first article.

```Python
covs[0]
```

--------------------------------

### Access Article Context

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Retrieves the context of the first article from the manager object.

```python
manager.articles[0].context
```

--------------------------------

### Custom Function to Add Multiple Columns

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Define and use a custom M function to add multiple columns dynamically. This function requires the table, a list of new column names, and a list of corresponding values. Ensure the number of columns and values match.

```M
AddMultipleColumns = (table as table, columns as list, values as list) => 
    let
        newtable = table,
        index = 0
    in
        for column in columns do
            newtable = Table.AddColumn(newtable, column, each values{index})
            index = index + 1
        newtable
in
    AddMultipleColumns
```

```M
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUzI1V...", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t]),
    #
```

```M
Added Multiple Columns" = AddMultipleColumns(Source, {"Column2", "Column3", "Column4"}, {"Value1", "Value2", "Value3"})
in
    #
```

```M
Added Multiple Columns"
```

--------------------------------

### Process and Aggregate Original Results

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Processes and aggregates results for the 'Original' method, iterating through tasks and data sources. It calculates average metrics and adds an overall average row to the DataFrame. Requires 'pandas'.

```python
def main(context_type):
    tasks = ['summarisation', 'qa', 'conversation']
    data_sources = ['news', 'arxiv']
    mask_ratios = [0.2, 0.35, 0.5, 0.65, 0.8]
    
    all_results = []
    method = f"Original"
    for task in tasks:
        if task == 'conversation':
            avg_results = read_results(tasks=['continue_conversation'], data_sources=['conversation'], mask_ratios=mask_ratios, context_type=context_type)
        else:
            avg_results = read_results(tasks=[task], data_sources=data_sources, mask_ratios=mask_ratios, context_type=context_type)
        avg_results['Method'] = method
        avg_results['Task'] = task
        avg_r = avg_results.to_frame().T.set_index(['Method', 'Task'])
        print(avg_r)
        all_results.append(avg_r)
    df = pd.concat(all_results)
    print(df)
    ratio_avg = df.mean()
    ratio_avg.name
    df.loc[(method, 'avg'), :] = ratio_avg
    print(df, "^^^^^^^^^")
    # all_results.append(df)
    
    # df = pd.concat(all_results, axis=0)
    print(df)
    return df

```

```python
df2 = main('no2-phrase')
```

--------------------------------

### Display DataFrame

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Prints the current state of the DataFrame 'df' to the console.

```python
df
```

--------------------------------

### Access Compressed Conversation Structure

Source: https://context7.com/liyucheng09/selective_context/llms.txt

Iterates through the first three compressed conversations to print their IDs and the beginning of their compressed context.

```python
# Access conversation structure
for conv in compressed_convs[:3]:
    print(f"Conversation ID: {conv.id}")
    print(f"Compressed: {conv.context[:300]}...")
    print("---")
```

--------------------------------

### Import OS Module

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Imports the operating system module, typically used for interacting with the operating system like file path manipulation or environment variables.

```python
import os
```

--------------------------------

### Duplicate Column to Create New Columns

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Use Table.DuplicateColumn to create new columns that are exact copies of an existing column. Specify the original column and a list of new column names.

```M
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUzI1V...", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t]),
    #
```

```M
Duplicate Columns" = Table.DuplicateColumn(Source, "Column1", {"Column2", "Column3", "Column4"})
in
    #
```

```M
Duplicate Columns"
```

--------------------------------

### Read Lexical Units with Mask Level

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

This snippet demonstrates how to read lexical units from article data, specifying a 'phrase' mask level. Ensure the 'data' object and 'articles' attribute are properly defined.

```python
print(read_lexical_units(data.articles[2], mask_level = 'phrase'))
```

--------------------------------

### Add Multiple Columns from a Record Column

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Expand a record column to create multiple new columns. This is useful when your data contains nested records that need to be flattened.

```M
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUzI1V...", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t, Column3 = _t]),
    #
```

```M
Expanded Record Column" = Table.ExpandRecordColumn(Source, "Record Column", {"Column4", "Column5", "Column6"})
in
    #
```

```M
Expanded Record Column"
```

--------------------------------

### Import Pickle Module

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the pickle module, which is necessary for serializing and de-serializing Python object structures.

```python
import pickle
```

--------------------------------

### Import Pandas Library in Python

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Imports the `pandas` library, a fundamental tool for data manipulation and analysis in Python.

```python
import pandas as pd
```

--------------------------------

### Execute Data-wise Analysis

Source: https://github.com/liyucheng09/selective_context/blob/main/results/show_results.ipynb

Calls the `data_wise` function with the 'self-info-phrase' context type to perform and display data-wise aggregation of results. This is used to analyze performance across different data sources.

```python
data_wise('self-info-phrase')
```

--------------------------------

### Create Pandas DataFrame from Dictionary

Source: https://github.com/liyucheng09/selective_context/blob/main/results/parse_results.ipynb

Creates a pandas DataFrame from a dictionary. The `orient='columns'` argument specifies that the dictionary keys should be treated as column names.

```python
df = pd.DataFrame.from_dict(covs, orient='columns')
```