JsonMachine (halaxa/json-machine)

JsonMachine

https://github.com/halaxa/json-machine
Admin
A memory-efficient PHP library for parsing large JSON files or streams incrementally, offering...

Tokens:8,683
Snippets:37
Trust Score:7.6
Update:5 months ago
Show doc for...
Context Summary (auto-generated)
Raw
# JSON Machine

JSON Machine is an efficient, easy-to-use PHP library for parsing extremely large JSON files and streams with constant memory footprint. The library solves the common problem of memory exhaustion when using `json_decode()` on large JSON documents by implementing a streaming parser that processes JSON incrementally, loading only one item at a time. Built on PHP generators, it provides a simple foreach-based API that works seamlessly with files, streams, and HTTP responses, making it ideal for processing multi-gigabyte JSON documents without memory constraints.

The library supports advanced features including JSON Pointer navigation for parsing specific subtrees, recursive iteration for deeply nested structures, multiple decoder options, custom error handling, and integration with popular HTTP clients like GuzzleHttp and Symfony HttpClient. With O(2) memory complexity for streams and thorough test coverage, JSON Machine delivers production-ready performance for data processing, ETL pipelines, API consumption, and any scenario requiring efficient handling of large-scale JSON data.

## Parsing JSON Files with Items::fromFile()

Parse large JSON files with constant memory usage by iterating items one at a time.

```php
<?php

use JsonMachine\Items;

// Parse a large JSON file - only loads one item at a time
$users = Items::fromFile('500MB-users.json');

foreach ($users as $id => $user) {
    // Process each user individually
    echo "User: {$user->name}, Email: {$user->email}\n";
}

// Get arrays instead of objects using ExtJsonDecoder
use JsonMachine\JsonDecoder\ExtJsonDecoder;

$data = Items::fromFile('data.json', ['decoder' => new ExtJsonDecoder(true)]);

foreach ($data as $key => $item) {
    // $item is now an associative array
    print_r($item);
}
```

## Parsing JSON Strings with Items::fromString()

Process JSON strings already in memory with reduced memory footprint compared to json_decode().

```php
<?php

use JsonMachine\Items;

$jsonString = '{"apple": {"color": "red"}, "pear": {"color": "yellow"}}';

$fruits = Items::fromString($jsonString);

foreach ($fruits as $name => $data) {
    echo "{$name}: {$data->color}\n";
}

// Output:
// apple: red
// pear: yellow
```

## Parsing Subtrees with JSON Pointer

Target specific parts of a JSON document using JSON Pointer (RFC 6901) to iterate only the data you need.

```php
<?php

use JsonMachine\Items;

// JSON: {"results": {"apple": {"color": "red"}, "pear": {"color": "yellow"}}}
$fruits = Items::fromFile('fruits.json', ['pointer' => '/results']);

foreach ($fruits as $name => $data) {
    echo "{$name}: {$data->color}\n";
}

// Parse nested array values with wildcard
// JSON: {"results": [{"name": "apple", "color": "red"}, {"name": "pear", "color": "yellow"}]}
$colors = Items::fromFile('fruitsArray.json', ['pointer' => '/results/-/color']);

foreach ($colors as $key => $value) {
    echo "Color: {$value}\n";
    // Access current JSON pointer
    echo "Path: {$colors->getCurrentJsonPointer()}\n";
}

// Output:
// Color: red
// Path: /results/0/color
// Color: yellow
// Path: /results/1/color
```

## Parsing Multiple Subtrees

Iterate over multiple document sections efficiently by specifying multiple JSON Pointers.

```php
<?php

use JsonMachine\Items;

// JSON with berries and citruses arrays
$fruits = Items::fromFile('fruits.json', [
    'pointer' => ['/berries', '/citruses']
]);

foreach ($fruits as $key => $value) {
    echo "Fruit: {$value['name']}, Color: {$value['color']}\n";
    echo "Section: {$fruits->getCurrentJsonPointer()}\n";
}

// Items are iterated in document order regardless of pointer order
```

## Parsing Streams with Items::fromStream()

Process streaming JSON data from network responses or other stream resources.

```php
<?php

use JsonMachine\Items;

// Parse a stream resource
$stream = fopen('https://api.example.com/data.json', 'r');
$items = Items::fromStream($stream);

foreach ($items as $key => $item) {
    // Process streaming data as it arrives
    processItem($item);
}

fclose($stream);
```

## GuzzleHttp Integration

Parse JSON responses from GuzzleHttp without loading entire response into memory.

```php
<?php

use JsonMachine\Items;
use GuzzleHttp\Client;
use GuzzleHttp\Psr7\StreamWrapper;

$client = new Client();
$response = $client->request('GET', 'https://api.example.com/large-dataset');

// Convert Guzzle PSR-7 stream to PHP stream resource
$phpStream = StreamWrapper::getResource($response->getBody());

$items = Items::fromStream($phpStream, ['pointer' => '/data']);

foreach ($items as $key => $value) {
    // Process each item from the API response
    echo "Processing item {$key}\n";
}
```

## Symfony HttpClient Integration

Parse streaming responses from Symfony HttpClient with native iterator support.

```php
<?php

use JsonMachine\Items;
use Symfony\Component\HttpClient\HttpClient;

function httpClientChunks($responseStream) {
    foreach ($responseStream as $chunk) {
        yield $chunk->getContent();
    }
}

$client = HttpClient::create();
$response = $client->request('GET', 'https://api.example.com/data');
$jsonChunks = httpClientChunks($client->stream($response));

$items = Items::fromIterable($jsonChunks, ['pointer' => '/results']);

foreach ($items as $key => $value) {
    // Process streaming API response
    var_dump($key, $value);
}
```

## Recursive Iteration with RecursiveItems

Handle deeply nested JSON structures without loading entire objects into memory.

```php
<?php

use JsonMachine\RecursiveItems;

// JSON: [{"username": "user1", "friends": [{"username": "friend1"}, {"username": "friend2"}]}]
$users = RecursiveItems::fromFile('users.json');

foreach ($users as $user) {
    // $user is a RecursiveItems instance
    foreach ($user as $field => $value) {
        if ($field === 'friends') {
            // $value is also RecursiveItems
            foreach ($value as $friend) {
                // Convert to array when manageable
                $friendArray = $friend->toArray();
                echo "Friend: {$friendArray['username']}\n";
            }
        } elseif ($field === 'username') {
            echo "User: {$value}\n";
        }
    }
}
```

## RecursiveItems Array Access and advanceToKey()

Use array-like syntax and key navigation methods for convenient access to nested data.

```php
<?php

use JsonMachine\RecursiveItems;

$users = RecursiveItems::fromFile('users.json');

foreach ($users as $user) {
    // Array-like access (syntax sugar for advanceToKey)
    $friends = $user['friends']; // equivalent to $user->advanceToKey('friends')

    foreach ($friends as $friend) {
        $friendData = $friend->toArray();
        echo $friendData['username'] . "\n";
    }
}

// Chain array access
$users = RecursiveItems::fromFile('users.json');
$secondFriendUsername = $users[0]['friends'][1]['username'];
echo $secondFriendUsername; // 'friend2'
```

## Error Handling with ErrorWrappingDecoder

Skip malformed JSON items instead of terminating on syntax errors.

```php
<?php

use JsonMachine\Items;
use JsonMachine\JsonDecoder\DecodingError;
use JsonMachine\JsonDecoder\ErrorWrappingDecoder;
use JsonMachine\JsonDecoder\ExtJsonDecoder;

$items = Items::fromFile('data-with-errors.json', [
    'decoder' => new ErrorWrappingDecoder(new ExtJsonDecoder())
]);

foreach ($items as $key => $item) {
    // Check if key or item is a decoding error
    if ($key instanceof DecodingError) {
        echo "Error in key: {$key->getErrorMessage()}\n";
        echo "Raw JSON: {$key->getJsonValue()}\n";
        continue;
    }

    if ($item instanceof DecodingError) {
        echo "Error in item {$key}: {$item->getErrorMessage()}\n";
        echo "Raw JSON: {$item->getJsonValue()}\n";
        continue;
    }

    // Process valid item
    processItem($key, $item);
}
```

## PassThruDecoder for Custom Parsing

Receive raw JSON strings without automatic decoding for custom processing.

```php
<?php

use JsonMachine\Items;
use JsonMachine\JsonDecoder\PassThruDecoder;

$items = Items::fromFile('data.json', ['decoder' => new PassThruDecoder()]);

foreach ($items as $key => $jsonString) {
    // $key and $jsonString are raw JSON strings
    // Parse with custom logic or external libraries
    $decoded = customJsonParser($jsonString);
    processItem($decoded);
}
```

## Tracking Parsing Progress

Monitor progress through large JSON documents with debug mode enabled.

```php
<?php

use JsonMachine\Items;

$fileSize = filesize('huge-file.json');
$items = Items::fromFile('huge-file.json', ['debug' => true]);

foreach ($items as $name => $data) {
    $position = $items->getPosition();
    $progress = intval($position / $fileSize * 100);

    echo "Progress: {$progress}% ({$position} / {$fileSize} bytes)\n";

    // Process item
    processData($data);
}

// Note: getPosition() returns 0 when debug is disabled for performance
```

## Parsing Single Scalar Values

Extract individual scalar values from anywhere in a JSON document efficiently.

```php
<?php

use JsonMachine\Items;

// JSON: {"lastModified": "2012-12-12", "data": {...large data...}}
$items = Items::fromFile('metadata.json', ['pointer' => '/lastModified']);

foreach ($items as $key => $value) {
    // Parser stops after finding the value - very efficient
    echo "Last modified: {$value}\n"; // "2012-12-12"
    break;
}

// Shortcut to get single value
$items = Items::fromFile('metadata.json', ['pointer' => '/lastModified']);
$lastModified = iterator_to_array($items)['lastModified'];
```

## Custom Iterator from Iterable

Parse JSON from any iterable that produces JSON chunks.

```php
<?php

use JsonMachine\Items;

// Custom generator that yields JSON chunks
function customJsonSource() {
    yield '{"users": [';
    yield '{"id": 1, "name": "Alice"},';
    yield '{"id": 2, "name": "Bob"}';
    yield ']}';
}

$items = Items::fromIterable(customJsonSource(), ['pointer' => '/users']);

foreach ($items as $user) {
    echo "User {$user->id}: {$user->name}\n";
}

// Output:
// User 1: Alice
// User 2: Bob
```

## JSON Machine is a powerful tool for processing JSON data at scale, enabling developers to work with massive JSON documents, streaming APIs, and data pipelines without memory constraints. The library's primary use cases include ETL operations on large datasets, consuming paginated API responses, processing log files, importing/exporting data from databases, real-time data processing from webhooks, and analyzing multi-gigabyte JSON archives. By replacing `json_decode(file_get_contents())` with `Items::fromFile()`, applications immediately gain the ability to process JSON of any size with predictable, minimal memory usage.

Integration is straightforward across various scenarios: use `Items::fromFile()` for local files, `Items::fromStream()` for network streams and API responses, `Items::fromString()` when JSON is already in memory, and `RecursiveItems` when individual items are too large to load completely. The library works seamlessly with modern HTTP clients through simple adapters, supports flexible JSON Pointer-based navigation to target specific data sections, and provides robust error handling to skip malformed items while continuing processing. With no production dependencies beyond optional ext-json, JSON Machine delivers production-grade performance with a simple, intuitive API that feels natural to PHP developers.