### Install Dependencies for RAG Example

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Installs the 'lost-in-the-middle' dataset and the 'llmlingua' library. Ensure you are in the correct directory for the 'lost-in-the-middle' installation.

```python
# Install dependency.
## Lost in the middle
!git clone https://github.com/nelson-liu/lost-in-the-middle
!cd lost-in-the-middle && echo "xopen" > requirements.txt && pip install -e .
## LLMLingu
!pip install llmlingua
```

--------------------------------

### LLMLingua-2 Example Usage

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

This Python script demonstrates the basic usage of LLMLingua-2 for prompt compression. It requires the `llmlingua` library to be installed.

```python
from llmlingua import PromptCompressor

compressor = PromptCompressor(model_name="gpt-4")
compressed_prompt = compressor.compress_prompt(prompt, instructions, context, example)
print(compressed_prompt)
```

--------------------------------

### Install Model Training Packages

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/README.md

Install the necessary packages for training your own compressor on collected data. This includes scikit-learn and tensorboard.

```bash
pip install scikit-learn
pip install tensorboard
```

--------------------------------

### Install LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/README.md

Install the LLMLingua library using pip. This is the first step to begin using LLMLingua-2 experiments.

```bash
pip install llmlingua
```

--------------------------------

### Install LLM Lingua and Datasets

Source: https://github.com/microsoft/llmlingua/blob/main/examples/CoT.ipynb

Use this command to install the LLM Lingua library and the datasets package. Ensure you have pip installed.

```bash
!pip install llmlingua datasets
```

--------------------------------

### Setup LLMLingua PromptCompressor

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Retrieval.ipynb

Initializes the PromptCompressor from the LLMLingua library. This is the first step before using any of its functionalities.

```python
# Setup LLMLingua
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
```

--------------------------------

### Install LLMLingua and Dependencies

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Retrieval.ipynb

Install the LLMLingua library along with sentence-transformers, voyageai, and cohere. This is a prerequisite for using LLMLingua's features.

```python
!pip install llmlingua sentence_transformers voyageai cohere
```

--------------------------------

### Initialize LLM Lingua Prompt Compressor

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Sets up the LLM Lingua prompt compressor. Ensure the 'llmlingua' library is installed.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
```

--------------------------------

### Select Example from Dataset

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Extracts a specific example (instruction, demonstration, question, and answer) from the loaded dataset using its index.

```python
# select an example from NaturalQuestions
instruction, demonstration_str, question, answer = [
    datasets[23][key] for key in ["instruction", "demonstration", "question", "answer"]
]
```

--------------------------------

### LLMLingua Integration with LangChain

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

This example demonstrates how to integrate LLMLingua into LangChain for prompt compression within a Retrieval-Augmented Generation (RAG) pipeline. Ensure LangChain is installed.

```python
from langchain.retrievers.llmlingua import LLMLinguaRetriever

retriever = LLMLinguaRetriever(llm=llm, prompt_compressor=prompt_compressor)
retriever.get_relevant_documents(query)
```

--------------------------------

### Setup Environment for SecurityLingua Training

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/securitylingua/readme.md

Execute this bash script to set up the necessary environment for training a SecurityLingua model.

```bash
bash env_setup.sh
```

--------------------------------

### RAG Example with LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

This example illustrates how to use LLMLingua for prompt compression in a Retrieval-Augmented Generation (RAG) scenario. It involves setting up a retriever and compressing the prompt before querying.

```python
from llmlingua import PromptCompressor
from langchain.chat_models import ChatOpenAI
from langchain.retrievers import BM25Retriever

llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0.1)
compressor = PromptCompressor(llm=llm, 
                              MAX_tokens=1000, 
                              device='cpu')

retriever = BM25Retriever.from_documents(docs)
retriever.k = 10

query = "What is the impact of prompt compression on RAG?"
compressed_prompt = compressor.compress_prompt(query, docs)

response = llm.invoke(compressed_prompt)
print(response.content)
```

--------------------------------

### Using Quantized Models with LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/DOCUMENT.md

Initialize PromptCompressor with a quantized model like 'TheBloke/Llama-2-7b-Chat-GPTQ'. Requires 'optimum' and 'auto-gptq' to be installed.

```bash
pip install optimum auto-gptq
```

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor("TheBloke/Llama-2-7b-Chat-GPTQ", model_config={"revision": "main"})
```

--------------------------------

### Install OpenAI Library

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Install the OpenAI Python client library, version 0.28, which is required for interacting with OpenAI's API.

```bash
!pip install openai==0.28
```

--------------------------------

### Install Dependencies

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Installs the llmlingua and llama-index libraries. This is a prerequisite for using the RAG functionalities.

```python
!pip install llmlingua llama-index
```

--------------------------------

### LLMLingua-2 Integration with LlamaIndex

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

This example shows how to use LLMLingua-2 as a node postprocessor in LlamaIndex for prompt compression. This can help reduce costs and improve efficiency in RAG applications. Ensure LlamaIndex is installed.

```python
from llama_index.llms import OpenAI
from llama_index.node_postprocessors import LLMLinguaNodePostprocessor

node_postprocessor = LLMLinguaNodePostprocessor(llm=OpenAI(model="gpt-3.5-turbo"))
query_engine = index.as_query_engine(node_postprocessors=[node_postprocessor])
response = query_engine.query(query)
```

--------------------------------

### Install Data Collection Packages

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/README.md

Install the required packages for collecting your own data using GPT-4. This includes the openai library and spacy for natural language processing.

```bash
pip install openai==0.28
```

```bash
pip install spacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Select Example Data

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Extracts context, question, and answer from a dataset entry. Prepares an instruction and formats the question for the model.

```python
contexts, question, answer = [
    dataset[1][key] for key in ["context", "input", "answers"]
]
instruction = "Please complete the code given below."
question = question + "\n\nNext line of code:\n"
```

--------------------------------

### Initialize PromptCompressor with Quantized Llama-2 Model

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

Instantiate the PromptCompressor with a quantized model like 'TheBloke/Llama-2-7b-Chat-GPTQ'. Requires 'optimum' and 'auto-gptq' to be installed. Specify the model revision if needed.

```python
llm_lingua = PromptCompressor("TheBloke/Llama-2-7b-Chat-GPTQ", model_config={"revision": "main"})
```

--------------------------------

### Prepare and Send Prompt to OpenAI API

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Prepares a prompt with instructions and a question, then sends it to the OpenAI Completion API using the gpt-35-turbo-instruct engine. Ensure you have the 'openai' library installed and configured.

```python
# The response from original prompt
import json

instruction = "Please reference the following examples to answer the math question,"
prompt = instruction + prompt_complex + "\n\nQuestion: " + question

request_data = {
    "prompt": prompt,
    "max_tokens": 400,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
    "stop": "\n\n",
}
response = openai.Completion.create(
    engine="gpt-35-turbo-instruct",
    **request_data,
)
print(json.dumps(response, indent=4))
```

--------------------------------

### Initialize PromptCompressor with Phi-2 Model

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

Instantiate the PromptCompressor using the 'microsoft/phi-2' model. This is a common starting point for general prompt compression tasks.

```python
llm_lingua = PromptCompressor("microsoft/phi-2")
```

--------------------------------

### Select and Print GSM8K Example

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Selects a question and its corresponding answer from the GSM8K test set and prints them to the console. Assumes the 'gsm8k_test' dataset has been loaded.

```python
# select an example from GSM8K
question, answer = [gsm8k_test[2][key] for key in ["question", "answer"]]
# Ground-truth Answer
print("Question:", question)
print("Answer:", answer)
```

--------------------------------

### Configure OpenAI API Key

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Set up the OpenAI API key for authentication. This example also shows configuration for Azure OpenAI Service (AOAI) including base URL, API type, and version.

```python
# Using the OAI
import openai

openai.api_key = "<insert_openai_key>"

# or Using the AOAI
import openai

openai.api_key = "<insert_openai_key>"
openai.api_base = "<insert_openai_base>"
openai.api_type = "azure"
openai.api_version = "2023-05-15"
```

--------------------------------

### Compress Prompt using LLMLingua-2

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Compress the provided context using the LLMLingua-2 compressor. This example sets a compression rate of 0.33 and forces specific tokens to be retained.

```python
# 2000 Compression
compressed_prompt = llm_lingua.compress_prompt(
    context,
    rate=0.33,
    force_tokens=["!", ".", "?", "\n"],
    drop_consecutive=True,
)
```

--------------------------------

### Initialize SpectaculumView

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Initializes the SpectaculumView and sets up the SurfaceTexture listener for frame availability. This is a common setup for Spectaculum applications.

```java
            surfaceTexture.setOnFrameAvailableListener(SpectaculumView.this);
        }
    };

            mEffectEventListener.onEffectError(index, effect, e);
        }
    }


import android.app.Activity;
import android.graphics.BitmapFactory;
import android.view.Menu;
import android.view.MenuItem;
import android.view.SubMenu;
import android.view.View;
import android.view.ViewGroup;
import android.widget.Toast;
import java.util.ArrayList;
import java.util.List;
import net.protyposis.android.spectaculum.SpectaculumView;
import net.protyposis.android.spectaculum.effects.Parameter;
import net.protyposis.android.spectaculum.effects.ImmersiveSensorNavigation;
import net.protyposis.android.spectaculum.effects.ContrastBrightnessAdjustmentEffect;
import net.protyposis.android.spectaculum.effects.EffectException;
import net.protyposis.android.spectaculum.effects.FlowAbsSubEffect;
import net.protyposis.android.spectaculum.effects.QrMarkerEffect;
import net.protyposis.android.spectaculum.effects.Effect;
import net.protyposis.android.spectaculum.effects.FlipEffect;
import net.protyposis.android.spectaculum.effects.FlowAbsEffect;
import net.protyposis.android.spectaculum.effects.KernelBlurEffect;
import net.protyposis.android.spectaculum.effects.KernelEdgeDetectEffect;
import net.protyposis.android.spectaculum.effects.KernelEmbossEffect;
import net.protyposis.android.spectaculum.effects.KernelGaussBlurEffect;
import net.protyposis.android.spectaculum.effects.KernelSharpenEffect;
import net.protyposis.android.spectaculum.effects.NoEffect;
import net.protyposis.android.spectaculum.effects.SimpleToonEffect;
import net.protyposis.android.spectaculum.effects.SobelEffect;
import net.protyposis.android.spectaculum.effects.ImmersiveTouchNavigation;
import net.protyposis.android.spectaculum.effects.StackEffect;
import net.protyposis.android.spectaculum.effects.WatermarkEffect;
import net.protyposis.android.spectaculum.gles.GLUtils;
import net.protyposis.android.spectaculum.effects.ColorFilterEffect;
import net.protyposis.android.spectaculum.effects.ImmersiveEffect;
import net.protyposis.android.spectaculumdemo.testeffect.InterlaceEffect;
/*
 * Copyright 2014 Mario Guggenberger <mg@protyposis.net>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package net.protyposis.android.spectaculumdemo;




/**
 * Created by Mario on 18.07.2014.
 *
 * Helper class for easy effect handling in the various Spectaculum views in this demo.
 * Prov
```

--------------------------------

### Download Dataset and Prompt

Source: https://github.com/microsoft/llmlingua/blob/main/examples/CoT.ipynb

Loads the GSM8K dataset and downloads a hard prompt file. Ensure you have the 'datasets' library installed.

```python
# Download the original prompt and dataset
from datasets import load_dataset

!wget https://raw.githubusercontent.com/FranxYao/chain-of-thought-hub/main/gsm8k/lib_prompt/prompt_hardest.txt
prompt_complex = open("./prompt_hardest.txt").read()
gsm8k = load_dataset("gsm8k", "main")
gsm8k_test = gsm8k["test"]
```

--------------------------------

### Display Ground-truth Answer

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Prints the ground-truth answer for the selected example.

```python
# Ground-truth Answer
answer
```

--------------------------------

### Structured Prompt Compression with LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

Compresses text by splitting it into sections and applying compression rates. Use <llmlingua> tags for context segmentation with optional rate and compress parameters. This example demonstrates compressing a structured prompt with specific rates for different sections.

```python
structured_prompt = "<llmlingua, compress=False>Speaker 4:</llmlingua><llmlingua, rate=0.4> Thank you. And can we do the functions for content? Items I believe are 11, three, 14, 16 and 28, I believe.</llmlingua><llmlingua, compress=False>
Speaker 0:</llmlingua><llmlingua, rate=0.4> Item 11 is a communication from Council on Price recommendation to increase appropriation in the general fund group in the City Manager Department by $200 to provide a contribution to the Friends of the Long Beach Public Library. Item 12 is communication from Councilman Super Now. Recommendation to increase appropriation in the special advertising and promotion fund group and the city manager's department by $10,000 to provide support for the end of summer celebration. Item 13 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the city manager department by $500 to provide a donation to the Jazz Angels . Item 14 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.</llmlingua><llmlingua, compress=False>
Speaker 4:</llmlingua><llmlingua, rate=0.6> We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.</llmlingua>"
compressed_prompt = llm_lingua.structured_compress_prompt(structured_prompt, instruction="", question="", rate=0.5)
print(compressed_prompt['compressed_prompt'])
```

--------------------------------

### Select GSM8K Example

Source: https://github.com/microsoft/llmlingua/blob/main/examples/CoT.ipynb

Selects a specific question and answer pair from the GSM8K test set for demonstration.

```python
# select an example from GSM8K
question, answer = [gsm8k_test[2][key] for key in ["question", "answer"]]
```

--------------------------------

### Download Text File

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Download a text file from a URL using wget. This example downloads 'paul_graham_essay.txt'.

```bash
!wget "https://www.dropbox.com/s/f6bmb19xdg0xedm/paul_graham_essay.txt?dl=1" -O paul_graham_essay.txt
```

--------------------------------

### Load Dataset with Hugging Face

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Loads a dataset from the Hugging Face Hub. Ensure the 'datasets' library is installed.

```python
from datasets import load_dataset

dataset = load_dataset("THUDM/LongBench", "repobench-p", split="test")
```

--------------------------------

### Prepare and Send Prompt to OpenAI API

Source: https://github.com/microsoft/llmlingua/blob/main/examples/OnlineMeeting.ipynb

Constructs a prompt by joining contexts and a question, then sends it to the OpenAI ChatCompletion API using the 'gpt-4-32k' model. This snippet is used for generating responses from a language model based on provided context and a specific query. Ensure the `openai` library is installed and configured.

```python
# The response from original prompt, using GPT-4-32k
import json

prompt = "\n\n".join([contexts, question])

message = [
    {"role": "user", "content": prompt},
]

request_data = {
    "messages": message,
    "max_tokens": 500,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-4-32k",
    **request_data,
)
print(json.dumps(response, indent=4))
```

--------------------------------

### Compress Prompt and Call OpenAI API

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Compresses a prompt using LLM Lingua with dynamic context compression and then sends the compressed prompt to the OpenAI ChatCompletion API. Ensure you have the 'llm-lingua' and 'openai' libraries installed and configured.

```python
compressed_prompt = llm_lingua.compress_prompt(
    contexts_list,
    instruction=instruction,
    question=question,
    target_token=2000,
    condition_compare=True,
    condition_in_question="after",
    rank_method="longllmlingua",
    use_sentence_level_filter=False,
    context_budget="+100",
    dynamic_context_compression_ratio=0.4,  # enable dynamic_context_compression_ratio
    reorder_context="sort",
)
message = [
    {"role": "user", "content": compressed_prompt["compressed_prompt"]},
]

request_data = {
    "messages": message,
    "max_tokens": 100,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-4-32k",
    **request_data,
)

print(json.dumps(compressed_prompt, indent=4))
print("Response:", response)
```

--------------------------------

### Compress Prompt and Get OpenAI Response

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Compresses a prompt using LLM Lingua with dynamic context compression and then sends it to OpenAI's ChatCompletion API. Requires `llm_lingua` and `openai` libraries. The `compress_prompt` function takes various parameters to control the compression process.

```python
compressed_prompt = llm_lingua.compress_prompt(
    demonstration_str.split("\n"),
    instruction=instruction,
    question=question,
    target_token=500,
    condition_compare=True,
    condition_in_question="after",
    rank_method="longllmlingua",
    use_sentence_level_filter=False,
    context_budget="+100",
    dynamic_context_compression_ratio=0.4,  # enable dynamic_context_compression_ratio
    reorder_context="sort",
)
message = [
    {"role": "user", "content": compressed_prompt["compressed_prompt"]},
]

request_data = {
    "messages": message,
    "max_tokens": 100,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-3.5-turbo",
    **request_data,
)

print(json.dumps(compressed_prompt, indent=4))
print("Response:", response)
```

--------------------------------

### Generate Response with GPT-4-32k

Source: https://github.com/microsoft/llmlingua/blob/main/examples/OnlineMeeting.ipynb

Constructs a prompt by joining context and question, then sends it to the GPT-4-32k model via the OpenAI API to get a response. The output is printed as a JSON string.

```python
import json

prompt = "\n\n".join([contexts, question])

messag e = [
    {"role": "user", "content": prompt},
]

request_data = {
    "messages": message,
    "max_tokens": 100,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-4-32k",
    **request_data,
)
print(json.dumps(response, indent=4))
```

--------------------------------

### Install Lost in the Middle Dependency

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Retrieval.ipynb

Installs the 'lost-in-the-middle' library from GitHub. This is a prerequisite for running retrieval examples related to the 'lost in the middle' phenomenon.

```python
# Install dependency.
## Lost in the middle
!git clone https://github.com/nelson-liu/lost-in-the-middle
!cd lost-in-the-middle && echo "xopen" > requirements.txt && pip install -e .
```

--------------------------------

### Compress Prompt and Get OpenAI Chat Completion

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Compresses a prompt using LLM Lingua and then sends it to OpenAI's ChatCompletion API. Use this when you need to reduce the token count of your prompts for cost or efficiency. Ensure you have the 'llm-lingua' and 'openai' libraries installed and configured.

```python
compressed_prompt = llm_lingua.compress_prompt(
    demonstration_str.split("\n"),
    instruction=instruction,
    question=question,
    target_token=100,
    condition_compare=True,
    condition_in_question="after",
    rank_method="longllmlingua",
    use_sentence_level_filter=False,
    context_budget="+100",
    dynamic_context_compression_ratio=0.4,  # enable dynamic_context_compression_ratio
    reorder_context="sort",
)
message = [
    {"role": "user", "content": compressed_prompt["compressed_prompt"]},
]

request_data = {
    "messages": message,
    "max_tokens": 100,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-3.5-turbo",
    **request_data,
)

print(json.dumps(compressed_prompt, indent=4))
print("Response:", response)
```

--------------------------------

### Load and Print Dataset Sample

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

This code demonstrates how to load a specific task dataset from the THUDM/LongBench dataset using the `datasets` library and print the first sample's reference answers. It's useful for inspecting dataset contents.

```python
task = "triviaqa"
dataset = load_dataset("THUDM/LongBench", task, split="test")
sample = dataset[0]
context = sample["context"]
reference = sample["answers"]
print(reference)
```

--------------------------------

### Initialize LLMLingua PromptCompressor

Source: https://github.com/microsoft/llmlingua/blob/main/DOCUMENT.md

Initialize the PromptCompressor for LLMLingua or LLMLingua-2. Specify the model name, device, and whether to use LLMLingua-2.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="NousResearch/Llama-2-7b-hf", # Default model, use "microsoft/llmlingua-2-xlm-roberta-large-meetingbank" or "microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank" for LLMLingua-2
    device_map="cuda",  # Device environment (e.g., 'cuda', 'cpu', 'mps')
    model_config={},
    open_api_config={},
    use_llmlingua2=False, # Whether to use llmlingua-2
)
```

--------------------------------

### Initialize LLMLingua-2 Compressor

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Instantiate the PromptCompressor with a specified model, enabling the LLMLingua-2 functionality.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)
```

--------------------------------

### Load Dataset Sample

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

This snippet shows how to load a specific task dataset from the LongBench benchmark using the `datasets` library and access a sample, printing its reference answer.

```python
task = "narrativeqa"
dataset = load_dataset("THUDM/LongBench", task, split="test")
sample = dataset[3]
context = sample["context"]
reference = sample["answers"]
print(reference)
```

--------------------------------

### Select MeetingBank Example

Source: https://github.com/microsoft/llmlingua/blob/main/examples/OnlineMeeting.ipynb

Extracts the 'source' field from the second entry (index 1) of the loaded MeetingBank dataset.

```python
contexts = dataset[1]["source"]
```

--------------------------------

### Prepare and Send Prompt to GPT-4

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Code.ipynb

Constructs a prompt using instruction, context, and question, then sends it to the GPT-4-32k model via the OpenAI API. Requires 'openai' and 'json' libraries.

```python
import json

prompt = "\n\n".join([instruction, contexts, question])

messages = [
    {"role": "user", "content": prompt},
]

request_data = {
    "messages": messages,
    "max_tokens": 100,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-4-32k",
    **request_data,
)
print(json.dumps(response, indent=4))
```

--------------------------------

### Initialize LLMLingua Postprocessor

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Instantiate the LongLLMLinguaPostprocessor with custom parameters for context compression. Configure instruction strings, target token limits, ranking methods, and additional compression arguments.

```python
from llama_index.query_engine import RetrieverQueryEngine
from llama_index.response_synthesizers import CompactAndRefine
from llama_index.indices.postprocessor import LongLLMLinguaPostprocessor

node_postprocessor = LongLLMLinguaPostprocessor(
    instruction_str="Given the context, please answer the final question",
    target_token=300,
    rank_method="longllmlingua",
    additional_compress_kwargs={
        "condition_compare": True,
        "condition_in_question": "after",
        "context_budget": "+100",
        "reorder_context": "sort",  # enable document reorder,
        "dynamic_context_compression_ratio": 0.3,
    },
)
```

--------------------------------

### Run Compression Script

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/README.md

Use this script to compress original context on various benchmarks. This is a preparatory step before evaluation.

```bash
bash evaluation/scripts/compress.sh
```

--------------------------------

### Initialize LongLLMLinguaPostprocessor for LlamaIndex

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Configure the LongLLMLinguaPostprocessor with specific instructions, target token count, and compression parameters. This is useful for optimizing RAG prompts by compressing context.

```python
from llama_index.query_engine import RetrieverQueryEngine
from llama_index.response_synthesizers import CompactAndRefine
from llama_index.indices.postprocessor import LongLLMLinguaPostprocessor

node_postprocessor = LongLLMLinguaPostprocessor(
    instruction_str="Given the context, please answer the final question",
    target_token=300,
    rank_method="longllmlingua",
    additional_compress_kwargs={
        "condition_compare": True,
        "condition_in_question": "after",
        "context_budget": "+100",
        "reorder_context": "sort",  # enable document reorder
        "dynamic_context_compression_ratio": 0.4, # enable dynamic compression ratio
    },
)
```

--------------------------------

### Load Dataset and Print Reference

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

This code snippet demonstrates how to load a specific task's test dataset from the THUDM/LongBench dataset and print the reference answers for the first sample. It's useful for setting up evaluation scenarios.

```python
task = "gov_report"
dataset = load_dataset("THUDM/LongBench", task, split="test")
sample = dataset[0]
context = sample["context"]
reference = sample["answers"]
print(reference)
```

--------------------------------

### Safeguard LLM with SecurityLingua

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/securitylingua/readme.md

Load the SecurityLingua model, compress a malicious prompt to reveal its intention, and construct an augmented system prompt to guide the LLM in detecting malicious behavior.

```python
from llmlingua import PromptCompressor
llm_lingua = PromptCompressor(
    model_name="SecurityLingua/securitylingua-xlm-s2s",
    use_slingua=True
)

# 1. compress the prompt to reveal the malicious intention
intention = llm_lingua.compress_prompt(malicious_prompt)

# 2. construct the augmented system prompt, to provide the LLM with the malicious intention
augmented_system_prompt = f"{system_prompt}\n\nTo help you better understand the user's intention to detect potential malicious behavior, I have extracted the user's intention and it is: {intention}. If you believe the user's intention is malicious, please donot respond or respond with I'm sorry, I can't help with that."

# at last, chat with the LLM using the augmented system prompt
response = vllm.generate([
    augmented_system_prompt + malicious_prompt
])
```

--------------------------------

### Initialize and Use SecurityLingua

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

Instantiate PromptCompressor for SecurityLingua using the 'SecurityLingua/securitylingua-xlm-s2s' model. Use this for analyzing malicious prompts to determine user intention.

```python
from llmlingua import PromptCompressor

securitylingua = PromptCompressor(
    model_name="SecurityLingua/securitylingua-xlm-s2s",
    use_slingua=True
)
intention = securitylingua.compress_prompt(malicious_prompt)
```

--------------------------------

### Calculate Reranker Results

Source: https://github.com/microsoft/llmlingua/blob/main/examples/Retrieval.ipynb

Processes a dataset to evaluate reranking methods. It loads data, generates prompts, and uses LLMLingua to get reranking indices. Results are logged to a CSV file.

```python
import json

from xopen import xopen
from copy import deepcopy
from tqdm import tqdm

from lost_in_the_middle.prompting import (
    Document,
    get_qa_prompt,
)


def get_reranker_results(rank_method):
    path = "lost-in-the-middle/qa_data/20_total_documents/nq-open-20_total_documents_gold_at_9.jsonl.gz"
    d_idx = 9
    res = []
    with xopen(path) as f:
        for ii, jj in tqdm(enumerate(f), total=2655):
            input_example = json.loads(jj)
            question = input_example["question"]
            documents = []
            for ctx in deepcopy(input_example["ctxs"]):
                documents.append(Document.from_dict(ctx))

            prompt = get_qa_prompt(
                question,
                documents,
                mention_random_ordering=False,
                query_aware_contextualization=False,
            )

            c = prompt.split("\n\n")
            instruction, question = c[0], c[-1]
            demonstration = "\n".join(c[1:-1])
            corpus = demonstration.split("\n")

            idx = llm_lingua.get_rank_results(
                corpus,
                question,
                rank_method,
                "none" if rank_method == "llmlingua" else "after",
                [0] * 20,
            )
            idx = [ii[0] for ii in idx].index(d_idx)
            res.append(idx)
    logs = [rank_method]
    for idx in range(1, 21):
        acc = len([ii for ii in res if ii < idx]) / len(res) * 100
        print(
            "R@{},{:.2f}".format(
                idx, len([ii for ii in res if ii < idx]) / len(res) * 100
            )
        )
        logs.append("{:.2f}".format(acc))
    with open("retrieval.csv", "a") as f:
        f.write(",".join(logs) + "\n")


for rank_method in [
    "bm25",
    "gzip",
    "sentbert",
    "openai",
    "bge",
    "bge_reranker",
    "bge_llmembedder",
    "jinza",
    "voyageai",
    "cohere",
    "llmlingua",
    "longllmlingua",
]:
    get_reranker_results(rank_method)
```

--------------------------------

### Create End-to-End RetrieverQueryEngine

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Initialize a RetrieverQueryEngine with a retriever and the LLMLingua postprocessor. This sets up the RAG pipeline for an end-to-end query process.

```python
retriever_query_engine = RetrieverQueryEngine.from_args(
    retriever, node_postprocessors=[node_postprocessor]
)
```

--------------------------------

### Configure Azure OpenAI API Key

Source: https://github.com/microsoft/llmlingua/blob/main/examples/CoT.ipynb

Sets up the Azure OpenAI API key, base URL, and version. Replace placeholders with your specific Azure details.

```python
# or Using the AOAI
import openai

openai.api_key = "<insert_openai_key>"
openai.api_base = "https://xxxx.openai.azure.com/"
openai.api_type = "azure"
openai.api_version = "2023-05-15"
```

--------------------------------

### Get Logprobs from OpenAI

Source: https://github.com/microsoft/llmlingua/blob/main/Transparency_FAQ.md

Use the OpenAI Completion API to retrieve log probabilities for input prompts. Ensure the model supports logprob retrieval and configure parameters like `logprobs` and `echo` accordingly.

```python
logp = openai.Completion.create(
    model="davinci-002",
    prompt="Please return the logprobs",
    logprobs=0,
    max_tokens=0,
    echo=True,
    temperature=0,
)
Out[3]:
<OpenAIObject text_completion id=-at > JSON: {
  "id": "",
  "object": "text_completion",
  "created": 1707295146,
  "model": "davinci-002",
  "choices": [
    {
      "text": "Please return the logprobs",
      "index": 0,
      "logprobs": {
        "tokens": [
          "Please",
          " return",
          " the",
          " log",
          "pro",
          "bs"
        ],
        "token_logprobs": [
          null,
          -6.9668007,
          -2.047512,
          -8.885729,
          -13.960022,
          -5.479665
        ],
        "top_logprobs": null,
        "text_offset": [
          0,
          6,
          13,
          17,
          21,
          24
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 6,
    "total_tokens": 6
  }
}
```

--------------------------------

### Run Model Training Script

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/README.md

Execute the provided shell script to train a compressor on the collected data. This script initiates the model training process.

```bash
bash model_training/train.sh
```

--------------------------------

### Download GSM8K Dataset and Prompt

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Downloads the 'prompt_hardest.txt' file and the GSM8K dataset from Hugging Face. Requires 'datasets' library and internet access.

```bash
!wget https://raw.githubusercontent.com/FranxYao/chain-of-thought-hub/main/gsm8k/lib_prompt/prompt_hardest.txt
```

```python
prompt_complex = open("./prompt_hardest.txt").read()
gsm8k = load_dataset("gsm8k", "main")
gsm8k_test = gsm8k["test"]
```

--------------------------------

### Compress Prompt and Call OpenAI Chat Completion

Source: https://github.com/microsoft/llmlingua/blob/main/examples/OnlineMeeting.ipynb

Compresses a prompt using LLM Lingua to fit a target token count and then sends it to OpenAI's ChatCompletion API. Ensure you have the `llm_lingua` and `openai` libraries installed and configured.

```python
compressed_prompt = llm_lingua.compress_prompt(
    contexts.split("\n"),
    instruction="",
    question=question,
    target_token=2000,
    condition_compare=True,
    condition_in_question="after",
    rank_method="longllmlingua",
    use_sentence_level_filter=False,
    context_budget="+100",
    dynamic_context_compression_ratio=0.4,  # enable dynamic_context_compression_ratio
    reorder_context="sort",
)
message = [
    {"role": "user", "content": compressed_prompt["compressed_prompt"]},
]

request_data = {
    "messages": message,
    "max_tokens": 500,
    "temperature": 0,
    "top_p": 1,
    "n": 1,
    "stream": False,
}
response = openai.ChatCompletion.create(
    "gpt-4-32k",
    **request_data,
)

print(json.dumps(compressed_prompt, indent=4))
print("Response:", response)
```

--------------------------------

### Train SecurityLingua Model (Multi-GPU)

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/securitylingua/readme.md

Perform multi-GPU training for the SecurityLingua model using the 'accelerate' library. This command launches the training script across multiple processes, specifying data and save paths, epochs, and Weights & Biases configurations.

```bash
ACCELERATE_LOG_LEVEL="ERROR" accelerate launch --num_processes 4 experiments/llmlingua2/model_training/train_roberta.py \
    --data_path experiments/llmlingua2/results/security_lingua/jailbreak_pairs_annotated_filtered.pt  \
    --save_path experiments/llmlingua2/results/models/xlm_slingua.pth \
    --num_epoch 5 \
    --run_name xlm_slingua \
    --wandb_project slingua \
    --wandb_name xlm_slingua
```

--------------------------------

### Integrate LLMLingua with LangChain Retriever

Source: https://github.com/microsoft/llmlingua/blob/main/Transparency_FAQ.md

Initialize LLMLinguaCompressor for use with LangChain's ContextualCompressionRetriever. Ensure necessary LangChain and OpenAI components are imported.

```python
from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.retrievers.document_compressors import LLMLinguaCompressor
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(temperature=0)

compressor = LLMLinguaCompressor(model_name="openai-community/gpt2", device_map="cpu")
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=retriever
)

compressed_docs = compression_retriever.get_relevant_documents(
    "What did the president say about Ketanji Jackson Brown"
)
pretty_print_docs(compressed_docs)
```

--------------------------------

### Load Dataset and Define Prompt Components

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

Load the MeetingBank dataset and extract the context, question, and reference answer. This prepares the data for prompt compression and evaluation.

```python
# Download the original prompt and dataset
from datasets import load_dataset

dataset = load_dataset("huuuyeah/meetingbank", split="test")
context = dataset[0]["transcript"]

question = "What is the agenda item three resolution 31669 about?\nAnswer:"
reference = "Encouraging individualized tenant assessment."
```

--------------------------------

### Configure OpenAI API for Azure

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Sets up the OpenAI client to use Azure OpenAI Service. Ensure you replace '<insert_openai_key>' with your actual API key.

```python
import openai

openai.api_key = "<insert_openai_key>"
openai.api_base = "https://xxxx.openai.azure.com/"
openai.api_type = "azure"
openai.api_version = "2023-05-15"
```

--------------------------------

### Retrieve Nodes and Initialize Synthesizer

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Perform initial retrieval of relevant nodes based on a question and set up the response synthesizer. This is the first step in a manual RAG pipeline composition.

```python
retrieved_nodes = retriever.retrieve(question)
synthesizer = CompactAndRefine()
```

--------------------------------

### Initialize LLMLingua-2 XLM-RoBERTa Compressor

Source: https://github.com/microsoft/llmlingua/blob/main/Transparency_FAQ.md

Initialize the LLMLingua-2 compressor using the XLM-RoBERTa large model. Set `use_llmlingua2=True` to enable LLMLingua-2 functionality. This model is suitable for meeting bank datasets.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True, # Whether to use llmlingua-2
)
compressed_prompt = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens = ['\n', '?'])
```

--------------------------------

### LLMLingua-2 Small Model Usage

Source: https://github.com/microsoft/llmlingua/blob/main/DOCUMENT.md

Initialize PromptCompressor for LLMLingua-2 using a smaller BERT-based model.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True,
)
```

--------------------------------

### Compress Prompt with LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/Transparency_FAQ.md

Compress a prompt using the LLMLingua library. Specify compression ratio, iterative size, and context budget. This is suitable for general prompt compression tasks.

```python
prompt  = compressor.compress_prompt(
    context=xxx,
    instruction=xxx,
    question=xxx,
    ratio=0.75,
    iterative_size=100,
    context_budget="*2",
)
```

--------------------------------

### Compress Prompt with LLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/examples/LLMLingua2.ipynb

This code demonstrates how to split a context into a list of passages and then use LLMLingua's compress_prompt function to reduce the token count to a target. It specifies force_tokens and uses context-level filtering.

```python
context_list = context.split("\nPassage:")
context_list = ["\nPassage:" + c for c in context_list]

# 2000 Compression
compressed_prompt = llm_lingua.compress_prompt(
    context_list,
    target_token=2000,
    force_tokens=["\nPassage:", ".", "?", "\n"],
    drop_consecutive=True,
    use_context_level_filter=True,
)
```

--------------------------------

### Analyze Context Compression

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAGLlamaIndex.ipynb

Compare the token counts of original and compressed contexts to evaluate the effectiveness of LLMLingua's compression. This helps in understanding the reduction ratio achieved.

```python
original_contexts = "\n\n".join([n.get_content() for n in retrieved_nodes])
compressed_contexts = "\n\n".join([n.get_content() for n in new_retrieved_nodes])

original_tokens = node_postprocessor._llm_lingua.get_token_length(original_contexts)
compressed_tokens = node_postprocessor._llm_lingua.get_token_length(compressed_contexts)

print(compressed_contexts)
print()
print("Original Tokens:", original_tokens)
print("Compressed Tokens:", compressed_tokens)
print("Compressed Ratio:", f"{original_tokens/(compressed_tokens + 1e-5):.2f}x")
```

--------------------------------

### Compress Prompt using LongLLMLingua

Source: https://github.com/microsoft/llmlingua/blob/main/README.md

Compress a list of prompts using LongLLMLingua with specific parameters for context management and compression ratio. Requires 'prompt_list' and 'question' variables to be defined.

```python
from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
compressed_prompt = llm_lingua.compress_prompt(
    prompt_list,
    question=question,
    rate=0.55,
    # Set the special parameter for LongLLMLingua
    condition_in_question="after_condition",
    reorder_context="sort",
    dynamic_context_compression_ratio=0.3, # or 0.4
    condition_compare=True,
    context_budget="+100",
    rank_method="longllmlingua",
)
```

--------------------------------

### Compress Data using GPT-4

Source: https://github.com/microsoft/llmlingua/blob/main/experiments/llmlingua2/data_collection/README.md

Initiate the compression process for your custom dataset using GPT-4. Ensure your data is formatted correctly and specify model parameters and save paths.

```bash
python compress.py --load_origin_from <your data path> \
--chunk_size 512 \
--compressor llmcomp \
--model_name gpt-4-32k \
--save_path <compressed data save path>
```

--------------------------------

### Configure OpenAI API Key

Source: https://github.com/microsoft/llmlingua/blob/main/examples/RAG.ipynb

Sets up the OpenAI API key for use with the OpenAI library. Replace '<insert_openai_key>' with your actual API key.

```python
# Using the OAI
import openai

openai.api_key = "<insert_openai_key>"
```