TOON for Laravel (sbsaga/toon)

TOON for Laravel

https://github.com/sbsaga/toon
Admin
TOON for Laravel is a token-efficient data format library that converts JSON and PHP arrays into...

Tokens:9,473
Snippets:51
Trust Score:5.7
Update:5 months ago
Show doc for...
Context Summary (auto-generated)
Raw
# TOON for Laravel

TOON (Token-Optimized Object Notation) is a Laravel package that transforms JSON and PHP arrays into a compact, human-readable data format optimized for AI prompts and LLM contexts. The package provides bidirectional conversion between standard JSON structures and TOON format, achieving up to 70% token reduction while preserving complete data fidelity. This makes it particularly valuable for developers working with ChatGPT, Claude, Gemini, Mistral, or OpenAI APIs where token efficiency directly impacts cost and performance.

The package integrates seamlessly with Laravel 9-12 through service providers, facades, and Artisan commands. It handles nested data structures, tabular rendering for uniform arrays, special character escaping, and type coercion during decoding. TOON maintains key order deterministically and provides human-readable output suitable for both machine processing and developer inspection.

## APIs and Functions

### Convert Data to TOON Format

Transform JSON strings, arrays, or objects into compact TOON notation using the main conversion method.

```php
use Sbsaga\Toon\Facades\Toon;

// Convert nested array with mixed data types
$data = [
    'user' => 'Sagar',
    'message' => 'Hello, how are you?',
    'priority' => 'high',
    'tasks' => [
        ['id' => 1, 'title' => 'Review PR', 'done' => false],
        ['id' => 2, 'title' => 'Deploy app', 'done' => true],
        ['id' => 3, 'title' => 'Update docs', 'done' => false],
    ],
    'metadata' => [
        'created_at' => '2025-01-15',
        'version' => 2.1
    ]
];

$toon = Toon::convert($data);
echo $toon;

/* Output:
user: Sagar
message: Hello\, how are you?
priority: high
tasks:
  items[3]{id,title,done}:
    1,Review PR,false
    2,Deploy app,true
    3,Update docs,false
metadata:
  created_at: 2025-01-15
  version: 2.1
*/

// Alternative: use encode() alias for semantic clarity
$toonEncoded = Toon::encode($data);
```

### Decode TOON Back to PHP Arrays

Convert TOON format strings back to native PHP arrays with automatic type coercion.

```php
use Sbsaga\Toon\Facades\Toon;

$toonString = <<<TOON
user: Alice
active: true
settings:
  theme: dark
  notifications: false
projects:
  items[2]{id,name,budget}:
    101,Website Redesign,50000
    102,Mobile App,75000.5
TOON;

$decoded = Toon::decode($toonString);
print_r($decoded);

/* Output:
Array
(
    [user] => Alice
    [active] => 1
    [settings] => Array
        (
            [theme] => dark
            [notifications] =>
        )
    [projects] => Array
        (
            [0] => Array
                (
                    [id] => 101
                    [name] => Website Redesign
                    [budget] => 50000
                )
            [1] => Array
                (
                    [id] => 102
                    [name] => Mobile App
                    [budget] => 75000.5
                )
        )
)
*/
```

### Estimate Token Usage

Calculate approximate token consumption for TOON-formatted strings to measure compression effectiveness.

```php
use Sbsaga\Toon\Facades\Toon;

$data = [
    'question' => 'Explain machine learning',
    'context' => 'User is a beginner',
    'model' => 'gpt-4',
    'temperature' => 0.7
];

$toon = Toon::convert($data);
$stats = Toon::estimateTokens($toon);

print_r($stats);

/* Output:
Array
(
    [words] => 8
    [chars] => 89
    [tokens_estimate] => 8
)
*/

// Compare with original JSON
$json = json_encode($data);
$jsonStats = Toon::estimateTokens($json);
$savings = round(100 - ($stats['tokens_estimate'] / $jsonStats['tokens_estimate'] * 100), 2);
echo "Token savings: {$savings}%\n";
```

### Laravel Controller Integration

Use TOON in Laravel controllers to optimize AI prompt payloads before sending to LLM APIs.

```php
namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Sbsaga\Toon\Facades\Toon;

class AiPromptController extends Controller
{
    public function optimizePrompt(Request $request)
    {
        // Fetch large dataset from database
        $users = \App\Models\User::with('orders')
            ->limit(100)
            ->get()
            ->toArray();

        // Convert to TOON for token-efficient AI context
        $toonContext = Toon::convert($users);
        $stats = Toon::estimateTokens($toonContext);

        // Send to OpenAI with optimized context
        $prompt = "Analyze these users:\n\n{$toonContext}";

        // Your OpenAI API call here
        // $response = OpenAI::chat()->create([...]);

        return response()->json([
            'toon_size' => strlen($toonContext),
            'json_size' => strlen(json_encode($users)),
            'token_estimate' => $stats['tokens_estimate'],
            'compression_ratio' => round((strlen($toonContext) / strlen(json_encode($users))) * 100, 2) . '%',
            'optimized_prompt' => $toonContext
        ]);
    }
}
```

### Benchmark Route Example

Create a route to compare JSON vs TOON compression performance.

```php
use Illuminate\Support\Facades\Route;
use Sbsaga\Toon\Facades\Toon;

Route::get('/toon-benchmark', function () {
    // Load test data
    $testData = [
        'users' => [],
        'timestamp' => now()->toISOString(),
        'metadata' => ['version' => '1.0', 'source' => 'api']
    ];

    // Generate sample users
    for ($i = 1; $i <= 50; $i++) {
        $testData['users'][] = [
            'id' => $i,
            'name' => "User {$i}",
            'email' => "user{$i}@example.com",
            'active' => $i % 2 === 0,
            'balance' => round($i * 123.45, 2)
        ];
    }

    $jsonEncoded = json_encode($testData, JSON_PRETTY_PRINT);
    $toonEncoded = Toon::convert($testData);

    $jsonStats = Toon::estimateTokens($jsonEncoded);
    $toonStats = Toon::estimateTokens($toonEncoded);

    return response()->json([
        'json_size_bytes' => strlen($jsonEncoded),
        'toon_size_bytes' => strlen($toonEncoded),
        'json_tokens' => $jsonStats['tokens_estimate'],
        'toon_tokens' => $toonStats['tokens_estimate'],
        'size_reduction_percent' => round(100 - (strlen($toonEncoded) / strlen($jsonEncoded) * 100), 2),
        'token_reduction_percent' => round(100 - ($toonStats['tokens_estimate'] / $jsonStats['tokens_estimate'] * 100), 2),
        'json_preview' => substr($jsonEncoded, 0, 200) . '...',
        'toon_preview' => substr($toonEncoded, 0, 200) . '...'
    ]);
});
```

### ToonConverter Direct Usage

Instantiate the converter class directly with custom configuration for fine-grained control.

```php
use Sbsaga\Toon\Converters\ToonConverter;

// Create converter with custom settings
$converter = new ToonConverter([
    'min_rows_to_tabular' => 3,      // Require at least 3 rows for table format
    'max_preview_items' => 50,        // Limit preview to 50 items
    'escape_style' => 'backslash'     // Use backslash escaping
]);

$data = [
    ['product' => 'Laptop', 'price' => 999.99, 'stock' => 15],
    ['product' => 'Mouse', 'price' => 29.99, 'stock' => 150],
    ['product' => 'Keyboard', 'price' => 79.99, 'stock' => 45],
    ['product' => 'Monitor', 'price' => 299.99, 'stock' => 8]
];

$toon = $converter->toToon($data);
echo $toon;

/* Output:
items[4]{product,price,stock}:
  Laptop,999.99,15
  Mouse,29.99,150
  Keyboard,79.99,45
  Monitor,299.99,8
*/

// Handle special characters with escaping
$message = ['note' => 'Hello, world: test\nNew line'];
$escaped = $converter->toToon($message);
// Output: note: Hello\, world\: test\nNew line
```

### ToonDecoder Direct Usage

Use the decoder class directly for converting TOON back to PHP with custom options.

```php
use Sbsaga\Toon\Converters\ToonDecoder;

$decoder = new ToonDecoder([
    'coerce_scalar_types' => true,    // Convert "true" → bool, "123" → int
    'escape_style' => 'backslash'
]);

$toonInput = <<<TOON
config:
  enabled: true
  max_connections: 100
  timeout: 30.5
  database: null
features:
  items[3]{name,enabled}:
    caching,true
    logging,false
    monitoring,true
TOON;

try {
    $decoded = $decoder->fromToon($toonInput);
    print_r($decoded);
} catch (\Sbsaga\Toon\Exceptions\ToonException $e) {
    echo "Parse error: " . $e->getMessage();
}

/* Output:
Array
(
    [config] => Array
        (
            [enabled] => 1
            [max_connections] => 100
            [timeout] => 30.5
            [database] =>
        )
    [features] => Array
        (
            [0] => Array
                (
                    [name] => caching
                    [enabled] => 1
                )
            [1] => Array
                (
                    [name] => logging
                    [enabled] =>
                )
            [2] => Array
                (
                    [name] => monitoring
                    [enabled] => 1
                )
        )
)
*/
```

### Artisan CLI Commands

Convert files between JSON and TOON formats using Artisan commands for batch processing and automation.

```bash
# Convert JSON file to TOON format
php artisan toon:convert storage/app/data.json --encode

# Convert TOON file back to JSON with pretty printing
php artisan toon:convert storage/app/data.toon --decode --pretty

# Save output to specific file
php artisan toon:convert storage/app/input.json --encode --output=storage/app/output.toon

# Read from STDIN (pipe input)
cat data.json | php artisan toon:convert --encode

# Use custom configuration file
php artisan toon:convert data.json --encode --config=config/custom-toon.php

# Decode with pretty JSON output to file
php artisan toon:convert data.toon --decode --pretty --output=result.json
```

### Configuration Publishing and Customization

Publish and customize TOON configuration for project-specific requirements.

```bash
# Publish configuration file
php artisan vendor:publish --provider="Sbsaga\Toon\ToonServiceProvider" --tag=config
```

```php
// config/toon.php
return [
    // Enable/disable TOON package globally
    'enabled' => env('TOON_ENABLED', true),

    // Character escaping method
    'escape_style' => env('TOON_ESCAPE_STYLE', 'backslash'),

    // Minimum rows needed for tabular rendering
    'min_rows_to_tabular' => env('TOON_MIN_ROWS_TO_TABULAR', 2),

    // Maximum items shown in previews
    'max_preview_items' => env('TOON_MAX_PREVIEW_ITEMS', 200),

    // Type coercion during decoding
    'coerce_scalar_types' => env('TOON_COERCE_SCALARS', true),
];
```

```bash
# .env file configuration
TOON_ENABLED=true
TOON_ESCAPE_STYLE=backslash
TOON_MIN_ROWS_TO_TABULAR=3
TOON_MAX_PREVIEW_ITEMS=100
TOON_COERCE_SCALARS=true
```

### Testing and Validation

Verify conversion accuracy with roundtrip testing to ensure data integrity.

```php
use Sbsaga\Toon\Converters\ToonConverter;
use Sbsaga\Toon\Converters\ToonDecoder;

// Original data structure
$original = [
    'users' => [
        ['id' => 1, 'name' => 'Alice', 'active' => true],
        ['id' => 2, 'name' => 'Bob', 'active' => false]
    ],
    'config' => [
        'timeout' => 30,
        'retry' => true
    ]
];

// Convert to TOON
$converter = new ToonConverter();
$toon = $converter->toToon($original);

// Decode back to array
$decoder = new ToonDecoder();
$decoded = $decoder->fromToon($toon);

// Validate roundtrip accuracy
if ($original === $decoded) {
    echo "✓ Roundtrip conversion successful\n";
} else {
    echo "✗ Data mismatch detected\n";
    print_r(['original' => $original, 'decoded' => $decoded]);
}

// Test with special characters
$specialChars = ['message' => "Line 1: test, value\nLine 2: more\\data"];
$toonSpecial = $converter->toToon($specialChars);
$decodedSpecial = $decoder->fromToon($toonSpecial);

assert($specialChars['message'] === $decodedSpecial['message'], 'Special char handling failed');
```

## Summary

TOON for Laravel serves as a powerful token optimization tool for applications integrating with large language models and AI services. Primary use cases include reducing API costs for OpenAI/Claude/Gemini interactions, preprocessing large datasets before sending to LLM contexts, logging AI conversations in storage-efficient formats, and optimizing prompt engineering workflows. The package excels in scenarios where token count directly impacts pricing or performance, such as batch processing user data for AI analysis, storing chat histories, or generating system prompts from database queries.

Integration patterns follow Laravel conventions through facade usage for quick conversions, service container resolution for dependency injection, Artisan commands for file processing pipelines, and direct class instantiation for custom configurations. The bidirectional conversion maintains data integrity while achieving 60-75% size reduction, making it particularly valuable for teams building AI-powered applications where prompt optimization and cost management are critical concerns. The human-readable format also serves as a debugging tool, allowing developers to inspect compact data structures without sacrificing comprehension.