### Clone Toolkit Repository Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/README.md Clones the prompt compression toolkit from its GitHub repository to your local machine. This is the first step to get started with the project. ```shell git clone https://github.com/3DAgentWorld/Toolkit-for-Prompt-Compression.git ``` -------------------------------- ### Run Compression and Evaluation Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/README.md Provides an example of how to use the toolkit for running compression tasks and evaluating performance. It loads a dataset, initializes a compressor, and runs the evaluation pipeline. ```python from pctoolkit.runners import run from pctoolkit.datasets import load_dataset from pctoolkit.metrics import load_metrics from pctoolkit.compressors import PromptCompressor compressor = PromptCompressor(type='SCCompressor', device='cuda') dataset_name = 'arxiv' dataset = load_dataset(dataset_name) run(compressor=compressor, dataset=dataset, metrics=load_metrics, ratio=0.1) ``` -------------------------------- ### Install scrl Library (Bash) Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/pctoolkit/methods/SCRL_new/README.md Steps to set up a Python environment and install the 'scrl' library for training, evaluation, and summarization. Requires Python 3.7/3.8. ```bash conda create -n my_env python=3.8 # or with venv: python3.8 -m venv conda activate my_env # or: source /bin/activate pip install -r requirements.txt pip install -e . ``` -------------------------------- ### Train a Model Example Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/GSM8K/README.md Basic example command for training a GPT2 sized model. This is provided for illustrative purposes and was not used for experiments in the paper. ```bash python train.py ``` -------------------------------- ### Install Python Dependencies Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/README.md Installs all necessary Python packages listed in the `requirements.txt` file. This ensures the toolkit has all its dependencies met for proper execution. ```shell pip install -r requirements.txt ``` -------------------------------- ### File Architecture Example Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/pctoolkit/pretrain_models/README.md Illustrates the expected directory structure for pre-trained models within the toolkit. This structure helps organize downloaded models for use. ```shell pctoolkit/pretrained_models/newsroom-L11/ ``` -------------------------------- ### Sample from Model Example Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/GSM8K/README.md Basic example command for sampling from a trained model. The core calculator sampling logic is in `calculator.py:sample`, which is noted as inefficient due to lack of batch support and activation caching. ```bash python sample.py ``` -------------------------------- ### Navigate to Project Directory Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/README.md Changes the current working directory to the root of the cloned prompt compression toolkit. This is necessary before installing dependencies or running scripts. ```shell cd Toolkit-for-Prompt-Compression ``` -------------------------------- ### Ball Swapping Scenario 1 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_seven_objects.txt Illustrates a sequence of ball swaps between three people (Alice, Bob, Claire) and tracks the ball each person holds at each step. This helps in understanding state changes through pairwise exchanges. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a blue ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Claire and Alice swap balls. Then, Alice and Bob swap balls. Finally, Claire and Bob swap balls. At the end of the game, Bob has the Options: (A) yellow ball (B) blue ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: yellow, Bob: blue, Claire: pink. (1) Claire and Alice swap balls: Alice: pink, Bob: blue, Claire: yellow. (2) Alice and Bob swap balls: Alice: blue, Bob: pink, Claire: yellow. (3) Claire and Bob swap balls: Alice: blue, Bob: yellow, Claire: pink. At the end of the game, Bob has the yellow ball. So the answer is (A). ``` -------------------------------- ### Ball Swapping Scenario 1 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_five_objects.txt Illustrates a sequence of ball swaps between three people (Alice, Bob, Claire) and tracks the ball each person holds at each step. This helps in understanding state changes through pairwise exchanges. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a blue ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Claire and Alice swap balls. Then, Alice and Bob swap balls. Finally, Claire and Bob swap balls. At the end of the game, Bob has the Options: (A) yellow ball (B) blue ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: yellow, Bob: blue, Claire: pink. (1) Claire and Alice swap balls: Alice: pink, Bob: blue, Claire: yellow. (2) Alice and Bob swap balls: Alice: blue, Bob: pink, Claire: yellow. (3) Claire and Bob swap balls: Alice: blue, Bob: yellow, Claire: pink. At the end of the game, Bob has the yellow ball. So the answer is (A). ``` -------------------------------- ### Ball Swapping Scenario 1 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_three_objects.txt Illustrates a sequence of ball swaps between three people (Alice, Bob, Claire) and tracks the ball each person holds at each step. This helps in understanding state changes through pairwise exchanges. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a yellow ball, Bob has a blue ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Claire and Alice swap balls. Then, Alice and Bob swap balls. Finally, Claire and Bob swap balls. At the end of the game, Bob has the Options: (A) yellow ball (B) blue ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: yellow, Bob: blue, Claire: pink. (1) Claire and Alice swap balls: Alice: pink, Bob: blue, Claire: yellow. (2) Alice and Bob swap balls: Alice: blue, Bob: pink, Claire: yellow. (3) Claire and Bob swap balls: Alice: blue, Bob: yellow, Claire: pink. At the end of the game, Bob has the yellow ball. So the answer is (A). ``` -------------------------------- ### Socratic Dataset Example Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/GSM8K/README.md Illustrates the format of the Socratic dataset, which includes automatically generated 'Socratic subquestions' in bold before each step of the solution. ```APIDOC Socratic Dataset Example: A carnival snack booth made $50 selling popcorn each day. It made three times as much selling cotton candy. For a 5-day activity, the booth has to pay $30 rent and $75 for the cost of the ingredients. How much did the booth earn for 5 days after paying the rent and the cost of ingredients? **How much did the booth make selling cotton candy each day? ** The booth made $50 x 3 = $<<50*3=150>>150 selling cotton candy each day. **How much did the booth make in a day? ** In a day, the booth made a total of $150 + $50 = $<<150+50=200>>200. **How much did the booth make in 5 days? ** In 5 days, they made a total of $200 x 5 = $<<200*5=1000>>1000. **How much did the booth have to pay? ** The booth has to pay a total of $30 + $75 = $<<30+75=105>>105. **How much did the booth earn after paying the rent and the cost of ingredients? ** Thus, the booth earned $1000 - $105 = $<<1000-105=895>>895. Note: Socratic subquestions are in bold. Calculation annotations are within <<>>. ``` -------------------------------- ### Ball Swapping Scenario 2 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_seven_objects.txt Presents another ball swapping problem involving Alice, Bob, and Claire, with a different initial ball distribution and swap sequence. It details the step-by-step process to find the final ball held by Alice. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a white ball, Bob has a purple ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Bob and Alice swap balls. Then, Bob and Claire swap balls. Finally, Bob and Alice swap balls. At the end of the game, Alice has the Options: (A) white ball (B) purple ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: white, Bob: purple, Claire: pink. (1) Bob and Alice swap balls: Alice: purple, Bob: white, Claire: pink. (2) Bob and Claire swap balls: Alice: purple, Bob: pink, Claire: white. (3) Bob and Alice swap balls: Alice: pink, Bob: purple, Claire: white. At the end of the game, Alice has the pink ball. So the answer is (C). ``` -------------------------------- ### Ball Swapping Scenario 2 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_three_objects.txt Presents another ball swapping problem involving Alice, Bob, and Claire, with a different initial ball distribution and swap sequence. It details the step-by-step process to find the final ball held by Alice. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a white ball, Bob has a purple ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Bob and Alice swap balls. Then, Bob and Claire swap balls. Finally, Bob and Alice swap balls. At the end of the game, Alice has the Options: (A) white ball (B) purple ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: white, Bob: purple, Claire: pink. (1) Bob and Alice swap balls: Alice: purple, Bob: white, Claire: pink. (2) Bob and Claire swap balls: Alice: purple, Bob: pink, Claire: white. (3) Bob and Alice swap balls: Alice: pink, Bob: purple, Claire: white. At the end of the game, Alice has the pink ball. So the answer is (C). ``` -------------------------------- ### Ball Swapping Scenario 2 Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_five_objects.txt Presents another ball swapping problem involving Alice, Bob, and Claire, with a different initial ball distribution and swap sequence. It details the step-by-step process to find the final ball held by Alice. ```text Q: Alice, Bob, and Claire are playing a game. At the start of the game, they are each holding a ball: Alice has a white ball, Bob has a purple ball, and Claire has a pink ball. As the game progresses, pairs of players trade balls. First, Bob and Alice swap balls. Then, Bob and Claire swap balls. Finally, Bob and Alice swap balls. At the end of the game, Alice has the Options: (A) white ball (B) purple ball (C) pink ball A: Let's think step by step. (0) At the start: Alice: white, Bob: purple, Claire: pink. (1) Bob and Alice swap balls: Alice: purple, Bob: white, Claire: pink. (2) Bob and Claire swap balls: Alice: purple, Bob: pink, Claire: white. (3) Bob and Alice swap balls: Alice: pink, Bob: purple, Claire: white. At the end of the game, Alice has the pink ball. So the answer is (C). ``` -------------------------------- ### Partner Swapping Scenario Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_three_objects.txt Details a scenario from square dancing where dancers (Alice, Bob, Claire) switch partners (Lola, Rodrigo, Patrick) through a series of exchanges. The step-by-step solution tracks who Alice is dancing with at the end. ```text Q: Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Rodrigo, and Claire is dancing with Patrick. Throughout the song, the dancers often trade partners. First, Alice and Bob switch partners. Then, Claire and Bob switch partners. Finally, Bob and Alice switch partners. At the end of the dance, Alice is dancing with Options: (A) Lola (B) Rodrigo (C) Patrick A: Let's think step by step. (0) At the start: Alice: Lola, Bob: Rodrigo, Claire: Patrick. (1) Alice and Bob switch partners: Alice: Rodrigo, Bob: Lola, Claire: Patrick. (2) Claire and Bob switch partners: Alice: Rodrigo, Bob: Patrick, Claire: Lola. (3) Bob and Alice switch partners: Alice: Patrick, Bob: Rodrigo, Claire: Lola. At the end of the dance, Alice is dancing with Patrick. So the answer is (C). ``` -------------------------------- ### Basic Prompt Compression Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/README.md Demonstrates the basic usage of the `PromptCompressor` class for prompt compression. It initializes a compressor and applies it to a test prompt with a specified compression ratio. ```python from pctoolkit.compressors import PromptCompressor compressor = PromptCompressor(type='SCCompressor', device='cuda') test_prompt = "test prompt" ratio = 0.3 result = compressor.compressgo(test_prompt, ratio) print(result) ``` -------------------------------- ### Project Dependencies Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/pctoolkit/methods/SCRL_new/requirements.txt This snippet lists the essential Python packages and their specific versions required to set up and run the toolkit. These dependencies ensure compatibility and proper functionality of the project's components. ```Python transformers==4.11.3 datasets==1.14.0 sentence-transformers==2.1.0 rouge-score==0.0.4 nltk==3.6.5 ``` -------------------------------- ### Partner Swapping Scenario Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_five_objects.txt Details a scenario from square dancing where dancers (Alice, Bob, Claire) switch partners (Lola, Rodrigo, Patrick) through a series of exchanges. The step-by-step solution tracks who Alice is dancing with at the end. ```text Q: Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Rodrigo, and Claire is dancing with Patrick. Throughout the song, the dancers often trade partners. First, Alice and Bob switch partners. Then, Claire and Bob switch partners. Finally, Bob and Alice switch partners. At the end of the dance, Alice is dancing with Options: (A) Lola (B) Rodrigo (C) Patrick A: Let's think step by step. (0) At the start: Alice: Lola, Bob: Rodrigo, Claire: Patrick. (1) Alice and Bob switch partners: Alice: Rodrigo, Bob: Lola, Claire: Patrick. (2) Claire and Bob switch partners: Alice: Rodrigo, Bob: Patrick, Claire: Lola. (3) Bob and Alice switch partners: Alice: Patrick, Bob: Rodrigo, Claire: Lola. At the end of the dance, Alice is dancing with Patrick. So the answer is (C). ``` -------------------------------- ### Evaluate Hill Climbing Baseline (Bash) Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/pctoolkit/methods/SCRL_new/README.md Evaluates the hill climbing baseline summaries against reference datasets. Uses 'make' commands, specifying the output file from the hill climbing process. ```bash make hc-eval-google HC_OUTPUT=data/hc-outputs/hc.L11.google.jsonl make hc-eval-duc2004 HC_OUTPUT=data/hc-outputs/hc.L11.duc2004.jsonl make hc-eval-gigaword HC_OUTPUT=data/hc-outputs/hc.L8.gigaword.jsonl make hc-eval-broadcast HC_OUTPUT=data/hc-outputs/hc.P75.broadcast.jsonl make hc-eval-bnc HC_OUTPUT=data/hc-outputs/hc.P75.bnc.jsonl ``` -------------------------------- ### Produce Summaries with Hill Climbing Baseline (Bash) Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/pctoolkit/methods/SCRL_new/README.md Generates sentence summaries using a hill climbing baseline algorithm. Configurable with steps, target length, dataset, and output file. Supports CPU execution. ```bash python bin/run_hc.py \ --config config/hc.json \ --steps 10 \ --target-len 11 \ --dataset data/test-data/google.jsonl \ --output data/hc-outputs/example.jsonl \ --device cpu ``` -------------------------------- ### Partner Swapping Scenario Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/tracking_shuffled_objects_seven_objects.txt Details a scenario from square dancing where dancers (Alice, Bob, Claire) switch partners (Lola, Rodrigo, Patrick) through a series of exchanges. The step-by-step solution tracks who Alice is dancing with at the end. ```text Q: Alice, Bob, and Claire are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Rodrigo, and Claire is dancing with Patrick. Throughout the song, the dancers often trade partners. First, Alice and Bob switch partners. Then, Claire and Bob switch partners. Finally, Bob and Alice switch partners. At the end of the dance, Alice is dancing with Options: (A) Lola (B) Rodrigo (C) Patrick A: Let's think step by step. (0) At the start: Alice: Lola, Bob: Rodrigo, Claire: Patrick. (1) Alice and Bob switch partners: Alice: Rodrigo, Bob: Lola, Claire: Patrick. (2) Claire and Bob switch partners: Alice: Rodrigo, Bob: Patrick, Claire: Lola. (3) Bob and Alice switch partners: Alice: Patrick, Bob: Rodrigo, Claire: Lola. At the end of the dance, Alice is dancing with Patrick. So the answer is (C). ``` -------------------------------- ### Identify Octagon from SVG Path Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/geometric_shapes.txt Analyzes an SVG path element to identify it as an octagon. It breaks down the 'M' (moveto) and 'L' (lineto) commands, counting the segments to determine the number of sides. This example demonstrates identifying an eight-sided shape. ```svg ``` -------------------------------- ### Identify Kite from SVG Path Source: https://github.com/3dagentworld/toolkit-for-prompt-compression/blob/main/dataset/BBH/cot-prompts/geometric_shapes.txt Explains how to identify a kite from an SVG path by analyzing its 'M' and 'L' commands and calculating side lengths. This example shows a four-sided shape where adjacent sides have equal lengths, distinguishing it from a rectangle. ```svg ```