### Install toolformer-pytorch

Source: https://github.com/lucidrains/toolformer-pytorch/blob/main/README.md

Install the toolformer-pytorch library using pip.

```bash
pip install toolformer-pytorch
```

--------------------------------

### Teach Toolformer to Use Calendar API

Source: https://github.com/lucidrains/toolformer-pytorch/blob/main/README.md

Example of teaching a language model to use a Calendar API. This involves defining the tool, creating a prompt to guide the model, and then initializing and using the Toolformer class. The model is finetuned on filtered results of API calls.

```python
import torch
from toolformer_pytorch import Toolformer, PaLM

# simple calendar api call - function that returns a string

def Calendar():
    import datetime
    from calendar import day_name, month_name
    now = datetime.datetime.now()
    return f'Today is {day_name[now.weekday()]}, {month_name[now.month]} {now.day}, {now.year}.'

# prompt for teaching it to use the Calendar function from above

prompt = f"""
Your task is to add calls to a Calendar API to a piece of text.
The API calls should help you get information required to complete the text.
You can call the API by writing "[Calendar()]"
Here are some examples of API calls:
Input: Today is the first Friday of the year.
Output: Today is the first [Calendar()] Friday of the year.
Input: The president of the United States is Joe Biden.
Output: The president of the United States is [Calendar()] Joe Biden.
Input: [input]
Output: 
"""

data = [
    "The store is never open on the weekend, so today it is closed.",
    "The number of days from now until Christmas is 30",
    "The current day of the week is Wednesday."
]

# model - here using PaLM, but any nn.Module that returns logits in the shape (batch, seq, num_tokens) is fine

model = PaLM(
    dim = 512,
    depth = 2,
    heads = 8,
    dim_head = 64
).cuda()

# toolformer

toolformer = Toolformer(
    model = model,
    model_seq_len = 256,
    teach_tool_prompt = prompt,
    tool_id = 'Calendar',
    tool = Calendar,
    finetune = True
)

# invoking this will
# (1) prompt the model with your inputs (data), inserted into [input] tag
# (2) with the sampled outputs, filter out the ones that made proper API calls
# (3) execute the API calls with the `tool` given
# (4) filter with the specialized filter function (which can be used independently as shown in the next section)
# (5) fine-tune on the filtered results

filtered_stats = toolformer(data)

# then, once you see the 'finetune complete' message

response = toolformer.sample_model_with_api_calls("How many days until the next new years?")

# hopefully you see it invoke the calendar and utilize the response of the api call...

```

--------------------------------

### Autoregressive Text Generation with Sampling

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Use the `sample` function for autoregressive text generation. Supports cursor-based sampling, API token constraints, and automatic API start token selection. Requires a model, sequence length, prime sequence, and sampling parameters.

```python
import torch
from toolformer_pytorch import sample, PaLM

# Create model
model = PaLM(
    dim=512,
    num_tokens=20000,
    depth=4,
    heads=8,
    dim_head=64
).cuda()

# Basic sampling from a prime sequence
prime = torch.randint(0, 20000, (2, 50)).cuda()  # Batch of 2, 50 tokens each

generated = sample(
    model=model,
    seq_len=200,                    # Total sequence length to generate
    prime=prime,                    # Starting tokens
    temperature=0.9,                # Sampling temperature (0 = greedy)
    eos_token_id=2,                 # Stop at end-of-sequence token
    pad_id=0                        # Padding token ID
)
print(f"Generated shape: {generated.shape}")  # (2, 200)
```

--------------------------------

### Initialize Toolformer with Calendar Tool

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Demonstrates the initialization of the Toolformer class with a custom Calendar tool and configuration for the full training pipeline. This includes setting up the model, prompt, tool, and fine-tuning parameters.

```python
import torch
from toolformer_pytorch import Toolformer, PaLM

# Define a custom tool - a simple calendar API
def Calendar():
    import datetime
    from calendar import day_name, month_name
    now = datetime.datetime.now()
    return f'Today is {day_name[now.weekday()]}, {month_name[now.month]} {now.day}, {now.year}.'

# Create the prompt that teaches the model how to use the tool
prompt = """
Your task is to add calls to a Calendar API to a piece of text.
The API calls should help you get information required to complete the text.
You can call the API by writing "[Calendar()]"
Here are some examples of API calls:
Input: Today is the first Friday of the year.
Output: Today is the first [Calendar()] Friday of the year.
Input: The president of the United States is Joe Biden.
Output: The president of the United States is [Calendar()] Joe Biden.
Input: [input]
Output:
"""

# Training data - sentences that could benefit from date/time information
data = [
    "The store is never open on the weekend, so today it is closed.",
    "The number of days from now until Christmas is 30",
    "The current day of the week is Wednesday."
]

# Create the language model (PaLM architecture included)
model = PaLM(
    dim=512,
    depth=2,
    heads=8,
    dim_head=64,
    num_tokens=20000
).cuda()

# Initialize Toolformer with the model and tool configuration
toolformer = Toolformer(
    model=model,
    model_seq_len=256,
    teach_tool_prompt=prompt,
    tool_id='Calendar',
    tool=Calendar,
    filter_threshold=1.0,        # Minimum perplexity improvement required
    finetune=True,               # Enable automatic fine-tuning
    finetune_lr=1e-4,
    finetune_epochs=3,
    finetune_batch_size=16,
    prompt_batch_size=4
)

# Run the full pipeline: prompt -> sample -> filter -> execute -> filter -> finetune
filtered_stats = toolformer(data)
# Returns FilteredResults with num_passed, num_failed, filtered_tokens, etc.

print(f"Samples passed filtering: {filtered_stats.num_passed}")
print(f"Samples failed filtering: {filtered_stats.num_failed}")
```

--------------------------------

### Built-in Tool Usage

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Demonstrates the usage of pre-built tools like Calendar, Calculator, WikiSearch, and MT for various tasks.

```python
from toolformer_pytorch.tools import (
    Calendar,
    Calculator,
    WikiSearch,
    MT,
    WolframAlphaCalculator,
    google_search,
    bing_search
)
from toolformer_pytorch.prompts import (
    calendar_prompt,
    calculator_prompt,
    wikipedia_search_prompt,
    machine_translation_prompt
)

# Calendar - returns current date
date_string = Calendar()
print(date_string)  # "Today is Monday, January 15, 2024."

# Calculator - evaluates mathematical expressions
result = Calculator('400/1400')
print(result)  # 0.29

result = Calculator('25*4+10')
print(result)  # 110.0

# Wikipedia Search - retrieves relevant documents using ColBERTv2
documents = WikiSearch(
    input_query="What is machine learning?",
    k=5  # Number of documents to retrieve
)
for doc in documents[:2]:
    print(doc[:100] + "...")

# Machine Translation - translates to English using NLLB
translated = MT("Bonjour, comment allez-vous?")
print(translated)  # "Hello, how are you?"

# Optional: Wolfram Alpha (requires API key in WOLFRAM_ALPHA_APPID env var)
# answer = WolframAlphaCalculator("integrate x^2")
# print(answer)

# Optional: Google Search (requires GOOGLE_API_KEY and GOOGLE_CSE_ID env vars)
# results = google_search("PyTorch documentation", num_results=5)
# for r in results:
#     print(f"{r['title']}: {r['link']}")
```

--------------------------------

### Sample with API Call Constraints

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Demonstrates sampling with constraints on API calls, such as limiting to one call per sequence and auto-selecting API tokens.

```python
API_START_ID = 19998

generated_with_api = sample(
    model=model,
    seq_len=256,
    prime=prime,
    temperature=0.9,
    call_api_only_once=True,                    # Limit to one API call per sequence
    api_start_token_id=API_START_ID,
    auto_select_api_start_token_when_topk=True, # Auto-select API token if in top-k
    select_api_start_id_top_k=10                # Check top 10 tokens for API start
)
```

--------------------------------

### Sample Model with Calculator API Calls

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Demonstrates how to use the `sample_model_with_api_calls` method to generate text with automatic API invocations. This method handles the cycle of generation, API execution, and continued generation after receiving API responses.

```python
from toolformer_pytorch import Toolformer, PaLM

def Calculator(expression):
    """Simple calculator that evaluates expressions"""
    from operator import add, sub, mul, truediv
    operators = {'+': add, '-': sub, '*': mul, '/': truediv}
    if expression.isdigit():
        return float(expression)
    for c in operators.keys():
        left, op, right = expression.partition(c)
        if op in operators:
            return round(operators[op](Calculator(left), Calculator(right)), 2)

model = PaLM(dim=512, depth=4, heads=8, dim_head=64).cuda()

toolformer = Toolformer(
    model=model,
    model_seq_len=512,
    teach_tool_prompt="...",  # Calculator prompt
    tool_id='Calculator',
    tool=Calculator,
    finetune=False
)

# After training, generate text with automatic API calls
# The model will insert [Calculator(...)] calls and receive results
response = toolformer.sample_model_with_api_calls(
    prime="What is 25 * 4?",    # Can be string or tensor
    occurrence=1                 # Number of API calls to allow
)

```

--------------------------------

### Sample with Custom Cursor Positions

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Illustrates sampling by continuing generation from custom cursor positions within the sequence, useful after API responses.

```python
positions = torch.tensor([45, 48]).cuda()  # Different start positions per batch

generated_from_positions = sample(
    model=model,
    seq_len=256,
    prime=prime,
    positions=positions,  # Each sequence continues from different position
    temperature=0.7
)
```

--------------------------------

### Toolformer with Custom Prompt

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Shows how to integrate a Toolformer model with a specific tool and its corresponding prompt for fine-tuning.

```python
from toolformer_pytorch import Toolformer, PaLM

model = PaLM(dim=512, depth=2, heads=8, dim_head=64).cuda()

# Use built-in calculator prompt
toolformer = Toolformer(
    model=model,
    model_seq_len=256,
    teach_tool_prompt=calculator_prompt,  # Pre-defined prompt
    tool_id='Calculator',
    tool=Calculator,
    finetune=True
)
```

--------------------------------

### Execute Toolformer pipeline

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Demonstrates the full training pipeline including generation, filtering, API execution, and fine-tuning using the Toolformer class.

```python
import torch
from toolformer_pytorch import Toolformer, PaLM

def MyTool(x):
    return x * 2

prompt = """
Add [MyTool(n)] calls to double numbers.
Input: The value is 5.
Output: The value is [MyTool(5)] 10.
Input: [input]
Output:
"""

model = PaLM(dim=512, depth=2, heads=8, dim_head=64).cuda()

toolformer = Toolformer(
    model=model,
    model_seq_len=256,
    teach_tool_prompt=prompt,
    tool_id='MyTool',
    tool=MyTool,
    finetune=False  # Control fine-tuning manually
)

data = ["The number 7 doubled is 14.", "Half of 20 is 10."]

# Step 1: Generate API calls (prompt the model)
data_with_api_calls = toolformer(
    data,
    return_after_generating_api_calls=True
)
print("Generated:", data_with_api_calls)

# Step 2: Filter to keep only valid API calls (one per sequence)
filtered_data, filtered_api_calls = toolformer(
    data,
    return_after_filtering_api_calls=True
)
print("After filtering:", filtered_api_calls)

# Step 3: Execute API calls and get responses
filtered_data, api_calls, api_responses = toolformer(
    data,
    return_after_making_api_calls=True
)
print("With responses:", api_responses)

# Step 4: Filter by perplexity improvement
filtered_results = toolformer(
    data,
    return_after_filtering_by_api_response=True
)
print(f"Passed: {filtered_results.num_passed}, Failed: {filtered_results.num_failed}")

# Step 5: Manual fine-tuning on filtered results
if filtered_results.num_passed > 0:
    toolformer.finetune(filtered_results)

# Or use individual methods directly
generated = toolformer.generate_data_with_api_calls(data, temperature=0.9)

included, excluded = toolformer.filter_and_keep_only_first_api_call(
    data, generated, return_excluded=True
)

responses = toolformer.make_api_calls(included[1])

final_results = toolformer.filter_by_api_responses(
    included[0], included[1], responses
)
```

--------------------------------

### PaLM Model Implementation

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Shows how to instantiate and use the PaLM language model for generating logits and calculating language modeling loss.

```python
import torch
from toolformer_pytorch import PaLM

# Create a PaLM language model
model = PaLM(
    dim=512,              # Model dimension
    depth=6,              # Number of transformer layers
    num_tokens=20000,     # Vocabulary size
    dim_head=64,          # Dimension per attention head
    heads=8,              # Number of attention heads
    ff_mult=4             # Feedforward multiplier (ff_dim = dim * ff_mult)
).cuda()

# Forward pass: tokens -> logits
tokens = torch.randint(0, 20000, (4, 512)).cuda()  # Batch of 4, 512 tokens
logits = model(tokens)
print(f"Logits shape: {logits.shape}")  # (4, 512, 20000)

# Use for language modeling loss
labels = tokens[:, 1:]   # Shift right for next-token prediction
logits = logits[:, :-1]  # Remove last position

loss = torch.nn.functional.cross_entropy(
    logits.reshape(-1, 20000),
    labels.reshape(-1),
    ignore_index=0  # Ignore padding
)
print(f"Loss: {loss.item()}")

# Sampling usage
model.eval()
with torch.no_grad():
    # Get next token probabilities
    next_logits = model(tokens)[:, -1, :]  # Last position logits
    probs = torch.softmax(next_logits, dim=-1)
    next_token = torch.multinomial(probs, num_samples=1)
```

--------------------------------

### Invoke Tools with Text API Calls

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Use `invoke_tools` to parse text, execute registered functions with extracted parameters, and insert responses. Handles string, integer, and float parameters. Unregistered functions are ignored.

```python
from toolformer_pytorch import invoke_tools

# Define tool functions
def inc(i):
    """Increment a number"""
    return i + 1

def dec(i):
    """Decrement a number"""
    return i - 1

def Calculator(expression):
    """Evaluate mathematical expression"""
    return eval(expression)  # Simplified for example

def Calendar():
    """Return current date"""
    import datetime
    return datetime.datetime.now().strftime("%Y-%m-%d")

# Create a registry mapping function names to implementations
function_registry = {
    'inc': inc,
    'dec': dec,
    'Calculator': Calculator,
    'Calendar': Calendar
}

# Text with API call placeholders (format: [FunctionName(args)])
text = 'The result of incrementing 5 is [inc(5)] and decrementing 3 is [dec(3)]'

# Execute all API calls and insert results
result = invoke_tools(
    registry=function_registry,
    text=text,
    delimiter='→',           # Separates call from result
    api_start=' [',          # API call start marker
    api_stop=' ]'            # API call end marker
)
# Output: 'The result of incrementing 5 is [inc(5) → 6] and decrementing 3 is [dec(3) → 2]'

print(result)

# Unregistered functions are left unchanged
text_with_unknown = 'Call [unknown(42)] and [inc(10)]'
result = invoke_tools(function_registry, text_with_unknown)
# Output: 'Call [unknown(42)] and [inc(10) → 11]'
print(result)
```

--------------------------------

### Invoke Tools on Text

Source: https://github.com/lucidrains/toolformer-pytorch/blob/main/README.md

Utility function to find and execute API calls within a given text string. It uses a provided function registry to map function names to actual Python functions and replaces the API call placeholders with their results.

```python
from toolformer_pytorch import invoke_tools

def inc(i):
    return i + 1

def dec(i):
    return i - 1

function_registry = dict(
    inc = inc,
    dec = dec
)

text = 'make the following api calls: [inc(1)] and [dec(2)] and [ignored(3)]'

invoke_tools(function_registry, text)

# make the following api calls: [inc(1) → 2] and [dec(2) → 1] and [ignored(3)]

```

--------------------------------

### Detecting API Calls in Text

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Utility functions for checking the presence of API calls and ensuring only one API call per sequence.

```python
from toolformer_pytorch import has_api_calls, replace_all_but_first

# Check if text contains valid API calls
text1 = "The answer is [Calculator(2+2)] which equals 4"
text2 = "No API calls here"
text3 = "Multiple: [inc(1)] and [dec(5)] calls"

print(has_api_calls(text1))  # True
print(has_api_calls(text2))  # False
print(has_api_calls(text3))  # True

# Custom API markers
print(has_api_calls(
    "Using <api>Calculator(5)</api>",
    api_start=' <api>',
    api_stop='</api>'
))  # True
```

--------------------------------

### Filter API Responses with Perplexity Improvement

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Use `filter_tokens_with_api_response` to compute a fitness score for API calls, retaining only those that reduce text perplexity. Requires model, original tokens, tokens with and without API responses, and a filter threshold.

```python
import torch
from toolformer_pytorch import PaLM, filter_tokens_with_api_response

# Initialize model
model = PaLM(
    dim=512,
    num_tokens=20000,
    depth=2,
    heads=8,
    dim_head=64
).cuda()

# Define special token IDs for API boundaries
API_START_ID = 19998  # Token ID for "["
API_STOP_ID = 19999   # Token ID for "]"

# Prepare three versions of each sequence:
# 1. Original text without any API call
tokens = torch.randint(0, 20000, (10, 1024)).cuda()

# 2. Text with API call but no response: "[Calculator(2+2)]"
tokens_without_api_response = torch.randint(0, 20000, (10, 1024)).cuda()
tokens_without_api_response[:, 512] = API_START_ID
tokens_without_api_response[:, 522] = API_STOP_ID

# 3. Text with API call AND response: "[Calculator(2+2) → 4]"
tokens_with_api_response = torch.randint(0, 20000, (10, 1024)).cuda()
tokens_with_api_response[:, 512] = API_START_ID
tokens_with_api_response[:, 530] = API_STOP_ID  # Longer to include response

# Filter sequences based on perplexity improvement
filtered_results = filter_tokens_with_api_response(
    model=model,
    tokens=tokens,
    tokens_with_api_response=tokens_with_api_response,
    tokens_without_api_response=tokens_without_api_response,
    filter_threshold=1.0,       # Minimum improvement threshold
    api_start_token_id=API_START_ID,
    api_end_token_id=API_STOP_ID
)

# Access filtering results
print(f"Passed: {filtered_results.num_passed}")
print(f"Failed: {filtered_results.num_failed}")
print(f"Selected indices: {filtered_results.selected_indices}")
print(f"Filtered tokens shape: {filtered_results.filtered_tokens.shape}")

# Use filtered tokens for fine-tuning
training_data = filtered_results.filtered_tokens_with_api_response
```

--------------------------------

### Filter Tokens with API Response

Source: https://github.com/lucidrains/toolformer-pytorch/blob/main/README.md

Demonstrates filtering transformer output tokens based on whether they contain a valid API response. This is useful for evaluating and finetuning models that generate API calls.

```python
import torch

from toolformer_pytorch import (
    Toolformer,
    PaLM,
    filter_tokens_with_api_response
)

# model

palm = PaLM(
    dim = 512,
    num_tokens = 20000,
    depth = 2,
    heads = 8,
    dim_head = 64
).cuda()

# mock some tokens

mock_start_pos = 512
mock_api_call_length = 10
mock_api_start_id = 19998
mock_api_stop_id = 19999

tokens = torch.randint(0, 20000, (10, 1024)).cuda()
tokens_with_api_response = torch.randint(0, 20000, (10, 1024)).cuda()
tokens_without_api_response = torch.randint(0, 20000, (10, 1024)).cuda()

tokens_with_api_response[:, mock_start_pos] = mock_api_start_id
tokens_with_api_response[:, mock_start_pos + mock_api_call_length] = mock_api_stop_id

tokens_without_api_response[:, mock_start_pos] = mock_api_start_id
tokens_without_api_response[:, mock_start_pos + mock_api_call_length] = mock_api_stop_id

# filter

filtered_results = filter_tokens_with_api_response(
    model = palm,
    tokens = tokens,
    tokens_with_api_response = tokens_with_api_response,
    tokens_without_api_response = tokens_without_api_response,
    filter_threshold = 1.,
    api_start_token_id = mock_api_start_id,
    api_end_token_id = mock_api_stop_id
)

```

--------------------------------

### Filter API calls in text

Source: https://context7.com/lucidrains/toolformer-pytorch/llms.txt

Use these utility functions to ensure only the first API call remains in a string, which is a requirement for the filtering step.

```python
text_multiple = "First [inc(1)] then [dec(2)] and [mul(3)]"
text_single = replace_all_but_first(text_multiple)
print(text_single)  # "First [inc(1)] then  and "

# This is critical for the filtering step which requires exactly one API call
data_with_calls = [
    "Result: [Calculator(10/2)] = 5",
    "Values: [inc(1)] and [inc(2)] and [inc(3)]",  # Multiple calls
    "No calls here"
]

filtered = []
for text in data_with_calls:
    if has_api_calls(text):
        # Ensure only one API call for filtering
        text = replace_all_but_first(text)
        filtered.append(text)

print(filtered)
# ['Result: [Calculator(10/2)] = 5', 'Values: [inc(1)] and  and ']
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.