Symfony JsonStreamer (symfony/json-streamer)

Symfony JsonStreamer

https://github.com/symfony/json-streamer
Admin
The Symfony JsonStreamer component provides powerful methods to read and write data structures from...

Tokens:4,093
Snippets:13
Trust Score:9.3
Update:5 months ago
Show doc for...
Context Summary (auto-generated)
Raw
# Symfony JSON Streamer Component

## Introduction

The Symfony JSON Streamer component provides powerful methods to read and write data structures from and into JSON streams with memory-efficient processing. This experimental component is designed to handle large JSON datasets by streaming data incrementally rather than loading entire documents into memory. It leverages PHP's type system and Symfony's TypeInfo component to generate optimized stream processors at runtime, enabling high-performance serialization and deserialization of complex object graphs.

The component supports advanced features including custom value transformers for data conversion, property name mapping through attributes, generic type handling, DateTime conversions, lazy instantiation for streamed resources, and comprehensive error handling. It generates PHP code at runtime that is cached for subsequent uses, making repeated operations extremely fast while maintaining the flexibility to handle diverse data structures including scalars, collections, objects with nested properties, enums, and union types.

## APIs and Functions

### JsonStreamReader::create() - Create a JSON Stream Reader

Creates a new JSON stream reader instance with optional custom value transformers and cache directories for generated reader code and lazy ghost proxies.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\ValueTransformer\StringToDateTimeValueTransformer;
use Symfony\Component\TypeInfo\Type;

// Create a basic reader with default configuration
$reader = JsonStreamReader::create();

// Create a reader with custom configuration
$reader = JsonStreamReader::create(
    valueTransformers: [
        'custom_transformer' => new StringToDateTimeValueTransformer()
    ],
    streamReadersDir: '/tmp/my_app/stream_readers',
    lazyGhostsDir: '/tmp/my_app/lazy_ghosts'
);

// Example class to deserialize
class User
{
    public int $id;
    public string $name;
    public string $email;
}

// Read from a JSON string
$json = '{"id": 42, "name": "John Doe", "email": "john@example.com"}';
$user = $reader->read($json, Type::object(User::class));

echo $user->name; // Output: John Doe
```

### JsonStreamReader::read() - Read JSON into Typed Objects

Reads JSON input (string or resource) and converts it into strongly-typed PHP objects, collections, or scalars according to the specified type definition.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\TypeInfo\Type;

$reader = JsonStreamReader::create();

// Read scalar values
$boolean = $reader->read('true', Type::bool());
$integer = $reader->read('42', Type::int());
$nullValue = $reader->read('null', Type::nullable(Type::string()));

// Read arrays and lists
$array = $reader->read('[1, 2, 3, 4, 5]', Type::list(Type::int()));
$dict = $reader->read('{"key1": "value1", "key2": "value2"}', Type::dict(Type::string()));

// Read from a stream resource for large files
$fileHandle = fopen('/path/to/large-dataset.json', 'r');
try {
    $data = $reader->read($fileHandle, Type::list(Type::object(User::class)));
    foreach ($data as $user) {
        // Process users lazily without loading entire file
        processUser($user);
    }
} finally {
    fclose($fileHandle);
}

// Read nested collections
$json = '[{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]';
$users = $reader->read($json, Type::list(Type::object(User::class)));
```

### JsonStreamWriter::create() - Create a JSON Stream Writer

Creates a new JSON stream writer instance with optional custom value transformers and a cache directory for generated writer code.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\JsonStreamer\ValueTransformer\DateTimeToStringValueTransformer;

// Create a basic writer with default configuration
$writer = JsonStreamWriter::create();

// Create a writer with custom value transformers
$writer = JsonStreamWriter::create(
    valueTransformers: [
        'custom_date_transformer' => new DateTimeToStringValueTransformer()
    ],
    streamWritersDir: '/tmp/my_app/stream_writers'
);
```

### JsonStreamWriter::write() - Write Objects to JSON Stream

Writes PHP data structures to JSON format as a traversable, memory-efficient stream of string chunks that can be iterated or converted to a complete string.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

$writer = JsonStreamWriter::create();

class Product
{
    public int $id = 100;
    public string $name = 'Laptop';
    public float $price = 999.99;
}

$product = new Product();

// Write and convert to string immediately
$json = (string) $writer->write($product, Type::object(Product::class));
echo $json; // {"id":100,"name":"Laptop","price":999.99}

// Iterate through chunks for streaming output
$result = $writer->write($product, Type::object(Product::class));
foreach ($result as $chunk) {
    echo $chunk; // Stream output chunk by chunk
}

// Write collections
$products = [new Product(), new Product()];
$json = (string) $writer->write($products, Type::list(Type::object(Product::class)));

// Write with options
$dateTime = new \DateTimeImmutable('2024-11-20');
$json = (string) $writer->write(
    $dateTime,
    Type::object(\DateTimeImmutable::class),
    ['date_time_format' => 'Y-m-d']
);
```

### StreamedName Attribute - Map Property Names

Defines custom JSON property names that differ from the PHP property names using the StreamedName attribute.

```php
<?php

use Symfony\Component\JsonStreamer\Attribute\StreamedName;
use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

class ApiResponse
{
    #[StreamedName('@id')]
    public int $id;

    #[StreamedName('user_name')]
    public string $username;

    public string $email; // No attribute, uses property name
}

$reader = JsonStreamReader::create();
$writer = JsonStreamWriter::create();

// Read JSON with custom field names
$json = '{"@id": 123, "user_name": "johndoe", "email": "john@example.com"}';
$response = $reader->read($json, Type::object(ApiResponse::class));
echo $response->id; // 123
echo $response->username; // johndoe

// Write back with custom field names
$output = (string) $writer->write($response, Type::object(ApiResponse::class));
echo $output; // {"@id":123,"user_name":"johndoe","email":"john@example.com"}
```

### ValueTransformer Attribute - Custom Value Transformation

Apply custom transformations to property values during serialization and deserialization using the ValueTransformer attribute with callables or service IDs.

```php
<?php

use Symfony\Component\JsonStreamer\Attribute\ValueTransformer;
use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

class Product
{
    public int $id;

    #[ValueTransformer(
        streamToNative: fn(string $value) => strtoupper($value),
        nativeToStream: fn(string $value) => strtolower($value)
    )]
    public string $code;

    #[ValueTransformer(
        streamToNative: fn(string $value) => (int)($value / 100),
        nativeToStream: fn(int $value) => (string)($value * 100)
    )]
    public int $priceInDollars;
}

$reader = JsonStreamReader::create();
$writer = JsonStreamWriter::create();

// Reading transforms "abc" to "ABC"
$json = '{"id": 1, "code": "abc", "priceInDollars": "500"}';
$product = $reader->read($json, Type::object(Product::class));
echo $product->code; // ABC
echo $product->priceInDollars; // 5

// Writing transforms "XYZ" to "xyz"
$product->code = 'XYZ';
$product->priceInDollars = 10;
$output = (string) $writer->write($product, Type::object(Product::class));
echo $output; // {"id":1,"code":"xyz","priceInDollars":"1000"}
```

### ValueTransformerInterface - Implement Custom Transformers

Create reusable value transformer classes by implementing the ValueTransformerInterface for complex transformation logic.

```php
<?php

use Symfony\Component\JsonStreamer\ValueTransformer\ValueTransformerInterface;
use Symfony\Component\JsonStreamer\Attribute\ValueTransformer;
use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\TypeInfo\Type;

// Custom transformer implementation
class CurrencyTransformer implements ValueTransformerInterface
{
    public function transform(mixed $value, array $options = []): int
    {
        // Convert string like "$99.99" to cents: 9999
        $numeric = preg_replace('/[^0-9.]/', '', $value);
        return (int)((float)$numeric * 100);
    }

    public static function getStreamValueType(): Type
    {
        return Type::string();
    }
}

// Reverse transformer for writing
class CentsToStringTransformer implements ValueTransformerInterface
{
    public function transform(mixed $value, array $options = []): string
    {
        return '$' . number_format($value / 100, 2);
    }

    public static function getStreamValueType(): Type
    {
        return Type::string();
    }
}

class Order
{
    public int $id;

    #[ValueTransformer(streamToNative: CurrencyTransformer::class)]
    public int $totalInCents;
}

$reader = JsonStreamReader::create(
    valueTransformers: [
        CurrencyTransformer::class => new CurrencyTransformer(),
        CentsToStringTransformer::class => new CentsToStringTransformer()
    ]
);

$json = '{"id": 42, "totalInCents": "$149.99"}';
$order = $reader->read($json, Type::object(Order::class));
echo $order->totalInCents; // 14999 (cents)
```

### Generic Types - Handle Parameterized Collections

Read and write collections with specific element types using generic type annotations for type-safe operations on templated classes.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

class User
{
    public int $id;
    public string $name;
}

/**
 * @template T
 */
class PagedResult
{
    /** @var list<T> */
    public array $items = [];
    public int $total = 0;
    public int $page = 1;
}

$reader = JsonStreamReader::create();
$writer = JsonStreamWriter::create();

// Read generic collection with specific type parameter
$json = '{"items": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}], "total": 2, "page": 1}';
$result = $reader->read(
    $json,
    Type::generic(Type::object(PagedResult::class), Type::object(User::class))
);

foreach ($result->items as $user) {
    echo $user->name . "\n"; // Alice, Bob
}

// Write generic collection
$pagedResult = new PagedResult();
$pagedResult->items = [new User(), new User()];
$pagedResult->total = 2;
$output = (string) $writer->write(
    $pagedResult,
    Type::generic(Type::object(PagedResult::class), Type::object(User::class))
);
```

### DateTime Handling - Automatic Date Conversion

Built-in transformers automatically handle conversion between JSON strings and PHP DateTime objects with customizable formats.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\JsonStreamer\ValueTransformer\DateTimeToStringValueTransformer;
use Symfony\Component\TypeInfo\Type;

class Event
{
    public int $id;
    public \DateTimeImmutable $startDate;
    public \DateTimeImmutable $endDate;
    public string $title;
}

$reader = JsonStreamReader::create();
$writer = JsonStreamWriter::create();

// Read dates in RFC3339 format (default)
$json = '{"id": 1, "startDate": "2024-11-20T10:00:00+00:00", "endDate": "2024-11-20T12:00:00+00:00", "title": "Conference"}';
$event = $reader->read($json, Type::object(Event::class));
echo $event->startDate->format('Y-m-d'); // 2024-11-20

// Read dates with custom format
$json2 = '{"id": 2, "startDate": "2024-12-25", "endDate": "2024-12-26", "title": "Holiday"}';
$event2 = $reader->read(
    $json2,
    Type::object(Event::class),
    [DateTimeToStringValueTransformer::FORMAT_KEY => 'Y-m-d']
);

// Write dates with custom format
$event->startDate = new \DateTimeImmutable('2025-01-01');
$event->endDate = new \DateTimeImmutable('2025-01-02');
$output = (string) $writer->write(
    $event,
    Type::object(Event::class),
    [DateTimeToStringValueTransformer::FORMAT_KEY => 'Y-m-d']
);
echo $output; // {"id":1,"startDate":"2025-01-01","endDate":"2025-01-02","title":"Conference"}
```

### Nullable and Union Types - Handle Complex Types

Support for nullable properties and union types allows flexible data models that can handle multiple value types or null values.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

enum Status: int
{
    case ACTIVE = 1;
    case INACTIVE = 0;
}

class Profile
{
    public int $id;
    public ?string $bio = null;
    public ?Status $status = null;
    public string|int|null $metadata = null;
}

$reader = JsonStreamReader::create();
$writer = JsonStreamWriter::create();

// Read with null values
$json1 = '{"id": 1, "bio": null, "status": null, "metadata": null}';
$profile1 = $reader->read($json1, Type::object(Profile::class));
var_dump($profile1->bio); // NULL

// Read with mixed types in union property
$json2 = '{"id": 2, "bio": "Developer", "status": 1, "metadata": "text"}';
$profile2 = $reader->read($json2, Type::object(Profile::class));
echo $profile2->metadata; // "text"

$json3 = '{"id": 3, "bio": "Designer", "status": 1, "metadata": 42}';
$profile3 = $reader->read($json3, Type::object(Profile::class));
echo $profile3->metadata; // 42

// Write nullable values
$profile = new Profile();
$profile->id = 4;
$profile->bio = null;
$output = (string) $writer->write($profile, Type::object(Profile::class));
echo $output; // {"id":4,"bio":null,"status":null,"metadata":null}
```

### Stream Processing - Memory-Efficient Large File Handling

Process large JSON files efficiently by reading from file handles, enabling lazy loading and iteration without consuming excessive memory.

```php
<?php

use Symfony\Component\JsonStreamer\JsonStreamReader;
use Symfony\Component\JsonStreamer\JsonStreamWriter;
use Symfony\Component\TypeInfo\Type;

class LogEntry
{
    public string $timestamp;
    public string $level;
    public string $message;
}

$reader = JsonStreamReader::create(
    streamReadersDir: '/tmp/stream_readers',
    lazyGhostsDir: '/tmp/lazy_ghosts'
);

// Process large log file without loading everything into memory
$fileHandle = fopen('/var/log/application.json', 'r');
if ($fileHandle === false) {
    throw new \RuntimeException('Cannot open file');
}

try {
    // Read returns an iterable that yields items lazily
    $logs = $reader->read($fileHandle, Type::iterable(Type::object(LogEntry::class)));

    $errorCount = 0;
    foreach ($logs as $entry) {
        if ($entry->level === 'ERROR') {
            $errorCount++;
            echo "Error at {$entry->timestamp}: {$entry->message}\n";
        }

        // Memory is freed after each iteration
        if ($errorCount >= 10) {
            break; // Stop after finding 10 errors
        }
    }
} finally {
    fclose($fileHandle);
}

// Write large datasets efficiently
$writer = JsonStreamWriter::create();
$output = $writer->write(
    generateLargeDataset(), // Generator function
    Type::list(Type::object(LogEntry::class))
);

// Stream directly to output or file
foreach ($output as $chunk) {
    echo $chunk; // Or: fwrite($outputFile, $chunk);
}
```

## Summary

The Symfony JSON Streamer component is designed for applications that need to process large JSON datasets efficiently, such as data pipelines, ETL processes, API clients consuming paginated endpoints, log processors, and data migration tools. It excels in scenarios where memory consumption is a concern, as it can handle files of arbitrary size by streaming data incrementally. The component is particularly valuable for microservices that exchange complex data structures, batch processing systems, and any application requiring high-performance JSON serialization with strong type safety.

Integration with existing Symfony applications is straightforward through dependency injection, with support for cache warming to pre-generate optimized stream processors during deployment. The component works seamlessly with Symfony's type system and can be extended with custom value transformers for domain-specific data conversions. Common integration patterns include configuring cache directories for production environments, registering custom transformers as services, using attributes to map between API contracts and domain models, and combining with Symfony's HTTP client for efficient API consumption. The generated stream processors are cached and reused across requests, ensuring minimal overhead after the initial code generation.