Laravel Ingest

https://github.com/zappzerapp/laravel-ingest
Admin
A robust, configuration-driven ETL and data import framework for Laravel that handles CSV/Excel...
Tokens:48,809
Snippets:488
Trust Score:7.9
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install FTP/SFTP Flysystem Packages

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

Instructions to install the necessary league/flysystem packages for FTP and SFTP support.

```bash
composer require league/flysystem-ftp      # for FTP
composer require league/flysystem-sftp-v3  # for SFTP
```

--------------------------------

### Install Laravel Ingest

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Install the package via Composer, publish its configuration and migrations, and then run the migrations to set up the necessary database tables.

```bash
composer require zappzerapp/laravel-ingest

# Publish config & migrations
php artisan vendor:publish --provider="LaravelIngest\IngestServiceProvider"

# Create tables
php artisan migrate
```

--------------------------------

### Install Recommended Security Packages

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/security.md

Install additional security packages for Laravel to enhance your application's security posture.

```bash
# Additional security packages for Laravel
composer require spatie/laravel-security
composer require laravel/fortify
composer require spatie/laravel-activitylog
```

--------------------------------

### start()

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Starts a new import run. You can specify the importer slug, a payload (like a file path or URL), an optional user for auditing, and a flag for dry runs.

```APIDOC
## start()

### Description
Start a new import run.

### Method
`public static function start(string $importer, mixed $payload = null, ?Authenticatable $user = null, bool $isDryRun = false): IngestRun`

### Parameters
#### Path Parameters
- **$importer** (string) - Required - The registered slug of the importer
- **$payload** (mixed) - Optional - Source data: file path, `UploadedFile`, URL, or `null`
- **$user** (?Authenticatable) - Optional - User to associate with the run (for auditing)
- **$isDryRun** (bool) - Optional - If `true`, validate and transform without persisting

### Response
#### Success Response (IngestRun)
- **IngestRun** - The Eloquent model for the newly created run.

### Throws
- `DefinitionNotFoundException` - If importer slug is not registered
- `InvalidConfigurationException` - If importer config is invalid
- `SourceException` - If source cannot be read
- `FileProcessingException` - If file validation fails

#### Examples
```php
use LaravelIngest\Facades\Ingest;
use Illuminate\Support\Facades\Auth;

// Basic usage with file path
$run = Ingest::start('product-importer', 'imports/products.csv');

// With authenticated user
$run = Ingest::start('product-importer', 'imports/products.csv', Auth::user());

// With uploaded file
$run = Ingest::start('product-importer', $request->file('import'));

// Dry run (no data saved)
$run = Ingest::start('product-importer', 'imports/products.csv', null, isDryRun: true);

// For importers with predefined sources (FTP, scheduled URL)
$run = Ingest::start('daily-stock-sync');

// Check the result
echo "Import started with ID: {$run->id}";
echo "Status: {$run->status->value}";
```
```

--------------------------------

### Upload File and Start Run

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Starts a new ingest run from a file upload.

```APIDOC
## POST /upload/{importerSlug}

### Description
Starts a new ingest run from a file upload.

### Method
POST

### Endpoint
/api/v1/ingest/upload/{importerSlug}

### Parameters
#### Path Parameters
- **importerSlug** (string) - Required - The slug of the importer definition (e.g., `user-importer`).

#### Request Body
- **file** (file) - Required - The data file to import (e.g., CSV, XLSX).
- **dry_run** (boolean) - Optional - If `1` or `true`, performs a simulation without saving data.

### Success Response (202 Accepted)
The newly created IngestRunResource.

### Request Example
```bash
curl -X POST \
  -H "Authorization: Bearer <token>" \
  -F "file=@/path/to/data.csv" \
  https://myapp.com/api/v1/ingest/upload/user-importer
```
```

--------------------------------

### Transformation Pipeline Example

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/advanced-features.md

Apply a sequence of transformations to a field using a pipeline. This example trims whitespace, converts to a slug, and applies a default value.

```php
use LaravelIngest\Transformers\TrimTransformer;
use LaravelIngest\Transformers\SlugTransformer;
use LaravelIngest\Transformers\DefaultValueTransformer;

// Pipeline: Trim -> Slug -> Default
->mapAndTransform('title', 'slug', [
    new TrimTransformer(),
    new SlugTransformer(),
    new DefaultValueTransformer('untitled'),
])
```

--------------------------------

### Start and Benchmark Ingest

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Starts the Docker Compose environment and executes the benchmark ingest command.

```bash
docker compose up -d
docker compose exec app php artisan benchmark:ingest
```

--------------------------------

### Clone Demo Project

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Clone the demonstration repository to explore a complete working example of Laravel Ingest.

```bash
# Clone the demo
git clone https://github.com/zappzerapp/Laravel-Ingest-Demo.git
cd Laravel-Ingest-Demo
```

--------------------------------

### Upload File and Start Ingest Run

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Starts a new ingest run by uploading a data file for a specified importer. Supports optional dry run simulation. Requires an Authorization token.

```bash
curl -X POST \
  -H "Authorization: Bearer <token>" \
  -F "file=@/path/to/data.csv" \
  https://myapp.com/api/v1/ingest/upload/user-importer
```

--------------------------------

### Configure FTP/SFTP Filesystem Disk

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

Example configuration for an FTP disk in config/filesystems.php. This is required for using SourceType::FTP or SourceType::SFTP.

```php
// config/filesystems.php
'disks' => [
    // ...
    'erp_ftp' => [
        'driver' => 'ftp',
        'host' => env('FTP_HOST'),
        'username' => env('FTP_USERNAME'),
        'password' => env('FTP_PASSWORD'),
        // ... other options
    ],
],
```

--------------------------------

### Direct Handler Injection Example

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/custom-source-handlers.md

Demonstrates how to use a custom handler directly without relying on the SourceType enum. This provides more flexibility by instantiating and starting the import with the handler instance.

```php
use App\Ingest\Handlers\JsonArrayHandler;
use LaravelIngest\IngestManager;

// In a controller or service
$json = '[{"name": "Item 1", "email": "item1@example.com"}, {"name": "Item 2", "email": "item2@example.com"}]';

// Start the import with the custom handler
$ingestManager = app(IngestManager::class);
$run = $ingestManager->start('my-importer', $json);
```

--------------------------------

### Start and Monitor Queue Worker

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

To resolve 'Queue worker is not running' errors, ensure your queue worker is started and monitored. This shows basic start commands and a daemonized worker with retry options.

```bash
# Start
php artisan queue:work

# Monitor with Supervisord
php artisan queue:work --daemon --sleep=1 --tries=3
```

--------------------------------

### Synthetic Key Example

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Demonstrates creating a composite key at runtime using `beforeRow` and then using it with `keyedBy`. Ensure the synthetic column is a fillable attribute or a database column.

```php
IngestConfig::for(Product::class)
    ->map('store_id', 'store_id')
    ->map('sku', 'sku')
    ->map('name', 'name')
    ->beforeRow(function (array & $row) {
        $row['composite_key'] = $row['store_id'] . '|' . $row['sku'];
    })
    ->keyedBy('composite_key')
    ->onDuplicate(DuplicateStrategy::UPDATE);
```

--------------------------------

### Controller Integration for Starting Imports

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Handle incoming file uploads in a controller to start an import process. Includes validation for the file and dry run option, with specific exception handling for missing importers or file processing errors.

```php
namespace App\Http\Controllers;

use LaravelIngest\Facades\Ingest;
use LaravelIngest\Exceptions\FileProcessingException;
use LaravelIngest\Exceptions\DefinitionNotFoundException;
use Illuminate\Http\Request;

class ImportController extends Controller
{
    public function store(Request $request, string $importer)
    {
        $request->validate([
            'file' => 'required|file',
            'dry_run' => 'boolean',
        ]);

        try {
            $run = Ingest::start(
                importer: $importer,
                payload: $request->file('file'),
                user: $request->user(),
                isDryRun: $request->boolean('dry_run')
            );

            return response()->json([
                'message' => 'Import started',
                'run_id' => $run->id,
                'status' => $run->status->value,
            ]);
        } catch (DefinitionNotFoundException $e) {
            return response()->json(['error' => 'Importer not found'], 404);
        } catch (FileProcessingException $e) {
            return response()->json(['error' => $e->getMessage()], 422);
        }
    }
}
```

--------------------------------

### Run Docker Tests

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Commands to manage the Docker environment for testing the package. Includes starting Docker, running tests, and checking code coverage.

```bash
# Start Docker
composer docker:up

# Run Tests
composer docker:test

# Check Coverage
composer docker:coverage
```

--------------------------------

### Example JSON File Structure

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

This is an example of the expected JSON file structure for the JSON source type. It must contain a single top-level array where each element is treated as a row.

```json
[
  {
    "full_name": "John Doe",
    "user_email": "john@example.com",
    "is_admin": "yes"
  },
  {
    "full_name": "Jane Smith",
    "user_email": "jane@example.com",
    "is_admin": "no"
  }
]
```

--------------------------------

### Use Absolute Paths for Filesystem Sources

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

When encountering 'File not found' errors, ensure you are using absolute paths for your filesystem sources. This example shows how to construct an absolute path using `base_path()`.

```php
->fromSource(SourceType::FILESYSTEM, ['path' => base_path('storage/app/imports/file.csv')])
```

--------------------------------

### Initialize IngestConfig for a Model

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Required. Use this static method to start configuring the import process for a specific Eloquent model.

```php
IngestConfig::for(\App\Models\Product::class)
```

--------------------------------

### Set Appropriate File Size Limits

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/config-files.md

Set appropriate file size limits based on your needs by configuring the `INGEST_MAX_FILE_SIZE` environment variable. This example sets a 10MB limit.

```env
INGEST_MAX_FILE_SIZE=10485760  # 10MB for smaller imports
```

--------------------------------

### Run an Importer with a Specific File

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/running-imports.md

Use the `ingest:run` Artisan command to manually start an import process for a specified importer slug and source file. The `--file` option is required for filesystem sources.

```bash
php artisan ingest:run product-importer --file="imports/products.csv"
```

--------------------------------

### Programmatically Start an Import

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/running-imports.md

Initiate an import process directly from your application code using the `Ingest` Facade. This method is suitable for complex workflows or custom logic, allowing you to specify the importer slug, payload, and associated user.

```php
use LaravelIngest\Facades\Ingest;
use Illuminate\Support\Facades\Auth;

$slug = 'product-importer';
$filePath = 'imports/products.csv'; // Path on a configured disk
$user = Auth::user();

// Start the import
$ingestRun = Ingest::start(
    importer: $slug, 
    payload: $filePath, 
    user: $user,
    isDryRun: false
);

// $ingestRun is the Eloquent model for the newly created run.
echo "Started ingest run with ID: " . $ingestRun->id;
```

--------------------------------

### Start a New Import Run

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Initiate a new import process by specifying the importer slug and providing the data source. The payload can be a file path, an UploadedFile object, a URL, or null for importers with predefined sources. An optional user can be associated for auditing, and a dry run mode is available for validation without persistence.

```php
use LaravelIngest\Facades\Ingest;
use Illuminate\Support\Facades\Auth;

// Basic usage with file path
$run = Ingest::start('product-importer', 'imports/products.csv');

// With authenticated user
$run = Ingest::start('product-importer', 'imports/products.csv', Auth::user());

// With uploaded file
$run = Ingest::start('product-importer', $request->file('import'));

// Dry run (no data saved)
$run = Ingest::start('product-importer', 'imports/products.csv', null, isDryRun: true);

// For importers with predefined sources (FTP, scheduled URL)
$run = Ingest::start('daily-stock-sync');

// Check the result
echo "Import started with ID: {$run->id}";
echo "Status: {$run->status->value}";
```

--------------------------------

### Implement Import Completion Listener Logic

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/events.md

In the `handle` method of your listener, access the `IngestRunCompleted` event payload to retrieve import details. This example shows how to notify the user associated with the import run.

```php
// app/Listeners/SendImportSummaryListener.php
namespace App\Listeners;

use LaravelIngest\Events\IngestRunCompleted;
use App\Notifications\ImportCompletedNotification;

class SendImportSummaryListener
{
    public function handle(IngestRunCompleted $event): void
    {
        $user = $event->ingestRun->user;

        if ($user) {
            $user->notify(new ImportCompletedNotification($event->ingestRun));
        }
    }
}
```

--------------------------------

### Implement Audit Logging for Import Events

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/security.md

Log import activities, including start and completion events, to maintain an audit trail. This helps in tracking who performed imports, when, and with what results.

```php
->beforeImport(function(UploadedFile $file) {
    // Log import attempt
    activity()
        ->causedBy(auth()->user())
        ->performedOn(new Product())
        ->withProperties([
            'file_name' => $file->getClientOriginalName(),
            'file_size' => $file->getSize(),
            'mime_type' => $file->getMimeType(),
            'ip_address' => request()->ip(),
        ])
        ->log('product_import_started');
})

->afterImport(function(ImportResult $result) {
    // Log import completion
    activity()
        ->causedBy(auth()->user())
        ->performedOn(new Product())
        ->withProperties([
            'total_rows' => $result->totalRows,
            'successful_rows' => $result->successfulRows,
            'failed_rows' => $result->failedRows,
            'duration' => $result->duration,
        ])
        ->log('product_import_completed');
})
```

--------------------------------

### Integrate Virus Scanning Before Import

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/security.md

Implement virus scanning for uploaded files, especially in production environments, to detect and block malware. This example shows integration with ClamAV.

```php
// In your Importer class
use Illuminate\Http\UploadedFile;

public function getConfig(): IngestConfig
{
    return IngestConfig::for(Product::class)
        ->beforeImport(function(UploadedFile $file) {
            // Integrate with virus scanning service
            if (app()->environment('production')) {
                $this->scanForMalware($file);
            }
        });
}

private function scanForMalware(UploadedFile $file): void
{
    // Example: ClamAV integration
    $clamav = new \Xenolope\Quahog\Client(
        new \Xenolope\Quahog\Socket('127.0.0.1', 3310)
    );
    
    $result = $clamav->scanFile($file->getPathname());
    
    if ($result->isInfected()) {
        throw new SecurityException('File contains malware');
    }
}
```

--------------------------------

### Implement Slack Notification Logic

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/events.md

Define the `toSlack` method within your notification class to format and send a success message to Slack. This example shows how to include import summary details in the message.

```php
// app/Notifications/ImportCompletedNotification.php
public function toSlack($notifiable)
{
    $run = $this->ingestRun;
    
    return (new SlackMessage)
        ->success()
        ->content("Import '{$run->importer_slug}' finished!")
        ->attachment(function ($attachment) use ($run) {
            $attachment->fields([
                'Total' => $run->total_rows,
                'Success' => $run->successful_rows,
                'Failed' => $run->failed_rows,
            ]);
        });
}
```

--------------------------------

### Define Ingest Source Handlers

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/config-files.md

Configure the available ingest source handlers by mapping keys to their respective class names. This example shows common handlers like 'upload', 'filesystem', 'ftp', 'sftp', 'url', and 'json-stream'.

```php
'upload' => LaravelIngest\Sources\UploadHandler::class,
        'filesystem' => LaravelIngest\Sources\FilesystemHandler::class,
        'ftp' => LaravelIngest\Sources\RemoteDiskHandler::class,
        'sftp' => LaravelIngest\Sources\RemoteDiskHandler::class,
        'url' => LaravelIngest\Sources\UrlHandler::class,
        'json-stream' => LaravelIngest\Sources\JsonHandler::class,
    ],
];
```

--------------------------------

### Scheduled Imports using Facade

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Schedule recurring import tasks using Laravel's scheduling capabilities. This example shows how to run specific importers at daily or hourly intervals.

```php
// routes/console.php or App\Console\Kernel

use LaravelIngest\Facades\Ingest;
use Illuminate\Support\Facades\Schedule;

Schedule::call(function () {
    Ingest::start('daily-stock-sync');
})->daily()->at('02:00');

Schedule::call(function () {
    Ingest::start('hourly-price-update');
})->hourly();
```

--------------------------------

### Configure Supervisor for Queue Worker

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

This is an example Supervisor configuration file for managing Laravel queue workers. It specifies process name, command, user, logging, and restart policies.

```ini
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/your/project/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/path/to/your/project/storage/logs/worker.log
stopwaitsecs=3600
```

--------------------------------

### Event-Driven Imports

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Process data files automatically when a specific event occurs, such as receiving a new data file. The listener uses the Ingest Facade to start the import based on event data.

```php
namespace App\Listeners;

use LaravelIngest\Facades\Ingest;
use App\Events\DataFileReceived;

class ProcessReceivedDataFile
{
    public function handle(DataFileReceived $event)
    {
        Ingest::start(
            importer: $event->importerSlug,
            payload: $event->filePath,
            user: $event->uploadedBy
        );
    }
}
```

--------------------------------

### API Endpoint for File Uploads (UPLOAD source)

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/running-imports.md

Starts an import process for importers configured with the `UPLOAD` source type. This requires a `multipart/form-data` POST request containing the file to be imported and an optional `dry_run` field.

```APIDOC
## POST /api/v1/ingest/upload/{importerSlug}

### Description
Triggers an import process for importers using the `UPLOAD` source type. This endpoint expects a `multipart/form-data` request containing the file to be imported.

### Method
POST

### Endpoint
`/api/v1/ingest/upload/{importerSlug}`

### Parameters
#### Path Parameters
- **importerSlug** (string) - Required - The slug of the importer to trigger.

#### Request Body
- **file** (file) - Required - The file to be imported.
- **dry_run** (boolean) - Optional - Set to `1` or `true` to simulate the import without saving data.

### Request Example
```bash
curl -X POST \
  -H "Authorization: Bearer <token>" \
  -F "file=@/path/to/users.csv" \
  -F "dry_run=1" \
  https://myapp.com/api/v1/ingest/upload/user-importer
```
```

--------------------------------

### Configure Ingest Source Type

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/enums.md

Examples of configuring the data source for an ingest process using the SourceType enum. Covers various sources like file uploads, filesystems, FTP, SFTP, URLs, and JSON streams.

```php
use LaravelIngest\Enums\SourceType;
use LaravelIngest\IngestConfig;

// File uploads (via API)
IngestConfig::for(User::class)
    ->fromSource(SourceType::UPLOAD)

// Local/S3 filesystem
IngestConfig::for(Product::class)
    ->fromSource(SourceType::FILESYSTEM, [
        'path' => 'imports/products.csv',
        'disk' => 's3',
    ])

// FTP server
IngestConfig::for(Order::class)
    ->fromSource(SourceType::FTP, [
        'disk' => 'ftp-server',
        'path' => 'exports/orders.csv',
    ])

// SFTP server
IngestConfig::for(Invoice::class)
    ->fromSource(SourceType::SFTP, [
        'disk' => 'sftp-server',
        'path' => 'data/invoices.csv',
    ])

// Remote URL
IngestConfig::for(Rate::class)
    ->fromSource(SourceType::URL, [
        'url' => 'https://api.example.com/rates.csv',
    ])

// JSON with streaming (memory efficient for large files)
IngestConfig::for(Event::class)
    ->fromSource(SourceType::JSON, [
        'path' => 'data/events.json',
        'pointer' => '/data/items', // JSON pointer to array
    ])
```

--------------------------------

### Applying Configurable Mappings

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Shows how to instantiate and configure the `ProductMapping` class using fluent methods before applying it to an ingest configuration.

```php
IngestConfig::for(Order::class)
    ->applyMapping(
        (new ProductMapping())->withSku(false)->withPriceDecimals(0),
        'item'
    );
```

--------------------------------

### Create Test CSV for Import

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

Prepare a small CSV file with sample data to test import functionality and debug issues without processing large datasets.

```csv
name,email,price
Test Product,test@example.com,19.99
Another Product,another@example.com,29.99
```

--------------------------------

### Get Ingest Run Status

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/monitoring-and-retries.md

Retrieve detailed information about an ingest run's status and progress by making a GET request to the show endpoint.

```APIDOC
## GET /api/v1/ingest/{ingestRun}

### Description
Retrieves the status and progress details of a specific ingest run.

### Method
GET

### Endpoint
`/api/v1/ingest/{ingestRun}`

### Parameters
#### Path Parameters
- **ingestRun** (integer) - Required - The ID of the ingest run to query.

### Response
#### Success Response (200)
- **IngestRun** (object) - Contains the full details of the ingest run, including processed rows if configured.

### Request Example
```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/1
```
```

--------------------------------

### Analyze Ingest Errors via API

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/monitoring-and-retries.md

Send a GET request to the `/api/v1/ingest/{ingestRun}/errors/summary` endpoint to get an aggregated summary of errors and their frequencies for failed ingest runs.

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/1/errors/summary
```

--------------------------------

### Publish Configuration Files

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/config-files.md

Use this Artisan command to publish the configuration files to your application.

```bash
php artisan vendor:publish --tag=ingest-config
```

--------------------------------

### Prepare CSV Data for Import

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/getting-started/first-importer.md

Create a CSV file with the specified columns that match the mapping defined in your importer configuration.

```csv
full_name,user_email,is_admin
John Doe,john@example.com,yes
Jane Smith,jane@example.com,no
```

--------------------------------

### Get Error Summary

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Retrieves an aggregated summary of errors for a specific ingest run.

```APIDOC
## GET /{ingestRun}/errors/summary

### Description
Retrieves an aggregated summary of errors for a specific ingest run. Useful for quickly understanding the most common issues without fetching all failed rows.

### Method
GET

### Endpoint
/api/v1/ingest/{ingestRun}/errors/summary

### Parameters
#### Path Parameters
- **ingestRun** (integer) - Required - The ID of the ingest run.

### Success Response (200 OK)
An IngestErrorSummaryResource object.

### Request Example
```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/123/errors/summary
```
```

--------------------------------

### Configurable Product Mapping Class

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Defines a reusable `ProductMapping` class with fluent methods for configuration, such as including SKU and setting price decimals. Implement `MappingInterface`.

```php
// app/Ingest/Mappings/ProductMapping.php
class ProductMapping implements MappingInterface
{
    private bool $includeSku = true;
    private ?int $priceDecimals = 2;

    public function apply(IngestConfig $config, string $prefix = ''): IngestConfig
    {
        $prefix = $prefix !== '' ? "{$prefix}_" : '';

        $config
            ->map("{$prefix}product_id", 'product_id')
            ->map("{$prefix}product_name", 'name')
            ->mapAndTransform(
                "{$prefix}price_cents",
                'price',
                new NumericTransformer(decimals: $this->priceDecimals)
            );

        if ($this->includeSku) {
            $config->map("{$prefix}sku", 'sku');
        }

        return $config;
    }

    public function withSku(bool $include = true): self
    {
        $this->includeSku = $include;
        return $this;
    }

    public function withPriceDecimals(int $decimals): self
    {
        $this->priceDecimals = $decimals;
        return $this;
    }
}
```

--------------------------------

### Run Import via Artisan

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Execute the import process using the Artisan command, specifying the importer name and the source file.

```bash
php artisan ingest:run user-importer --file=users.csv
```

--------------------------------

### Log Database Queries

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

Enable logging of all executed SQL queries in the `local` environment by adding a listener in `AppServiceProvider::boot()`. This helps in identifying inefficient queries.

```php
// In AppServiceProvider::boot()
if (app()->environment('local')) {
    DB::listen(function($query) {
        Log::info($query->sql, $query->bindings);
    });
}
```

--------------------------------

### Trigger Non-Upload Run

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Starts a new ingest run for a source that does not require a file upload (e.g., `FTP`, `URL`).

```APIDOC
## POST /trigger/{importerSlug}

### Description
Starts a new ingest run for a source that does not require a file upload (e.g., `FTP`, `URL`).

### Method
POST

### Endpoint
/api/v1/ingest/trigger/{importerSlug}

### Parameters
#### Path Parameters
- **importerSlug** (string) - Required - The slug of the importer definition.

### Success Response (202 Accepted)
The newly created IngestRunResource.

### Request Example
```bash
curl -X POST \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/trigger/daily-stock-importer
```
```

--------------------------------

### Analyze Ingest Errors

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/monitoring-and-retries.md

Get an aggregated summary of errors for runs with failed rows to quickly identify common issues.

```APIDOC
## GET /api/v1/ingest/{ingestRun}/errors/summary

### Description
Provides an aggregated summary of errors encountered during a specific ingest run.

### Method
GET

### Endpoint
`/api/v1/ingest/{ingestRun}/errors/summary`

### Parameters
#### Path Parameters
- **ingestRun** (integer) - Required - The ID of the ingest run to analyze errors for.

### Response
#### Success Response (200)
- **data** (object) - Contains error and validation summaries.
  - **total_failed_rows** (integer) - The total number of rows that failed.
  - **error_summary** (array) - A list of aggregated error messages and their counts.
    - **message** (string) - The error message.
    - **count** (integer) - The number of times this error occurred.
  - **validation_summary** (array) - A list of aggregated validation messages and their counts.
    - **message** (string) - The validation message.
    - **count** (integer) - The number of times this validation failed.

### Request Example
```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/1/errors/summary
```

### Response Example
```json
{
  "data": {
    "total_failed_rows": 15,
    "error_summary": [
      { "message": "Duplicate entry found for key 'email'.", "count": 8 }
    ],
    "validation_summary": [
      { "message": "email: The email field is required.", "count": 5 }
    ]
  }
}
```
```

--------------------------------

### Trigger Non-Upload Ingest Run

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Starts a new ingest run for sources that do not require file uploads, such as FTP or URL. Requires an Authorization token.

```bash
curl -X POST \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/trigger/daily-stock-importer
```

--------------------------------

### Configure Filesystem Source Type (Hardcoded Path)

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

Use SourceType::FILESYSTEM with a hardcoded path and optional disk for imports like nightly cron jobs. The 'path' option is required, and 'disk' can override the default.

```php
// IngestConfig for a hardcoded path (e.g. nightly cron)
->fromSource(SourceType::FILESYSTEM, ['path' => 'imports/daily-products.csv', 'disk' => 's3'])
```

--------------------------------

### Get All Importer Definitions

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Retrieve all registered importer definitions. This returns an associative array where keys are the importer slugs and values are the corresponding IngestDefinition instances.

```php
use LaravelIngest\Facades\Ingest;

$definitions = Ingest::getDefinitions();
```

--------------------------------

### Handle ConcurrencyException in Laravel Ingest

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/exceptions.md

Catch `ConcurrencyException` when concurrent operations conflict. This example shows handling the case where another retry is already in progress.

```php
use LaravelIngest\Exceptions\ConcurrencyException;

try {
    $run = Ingest::retry($originalRun);
} catch (ConcurrencyException $e) {
    // Another retry is already in progress
}
```

--------------------------------

### Monitor Ingest Run Status via API

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/monitoring-and-retries.md

Make a GET request to the `/api/v1/ingest/{ingestRun}` endpoint to retrieve the full `IngestRun` details as a JSON object.

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/1
```

--------------------------------

### Configure Ingest with Fluent API

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/README.md

Demonstrates advanced configuration of the IngestConfig fluent API for product imports. This includes setting the source, performance options, duplicate handling, and custom transformations.

```php
IngestConfig::for(Product::class)
    // Sources: UPLOAD, FILESYSTEM, URL, FTP, SFTP
    ->fromSource(SourceType::FTP, ['disk' => 'erp', 'path' => 'daily.csv'])
    
    // Performance
    ->setChunkSize(1000)
    ->atomic() // Wrap chunks in transactions
    
    // Logic
    ->keyedBy('sku')
    ->onDuplicate(DuplicateStrategy::UPDATE_IF_NEWER)
    ->compareTimestamp('last_modified_at', 'updated_at')
    
    // Transformation
    ->mapAndTransform('price_cents', 'price', fn($val) => $val / 100)
    ->resolveModelUsing(fn($row) => $row['type'] === 'digital' ? DigitalProduct::class : Product::class);
```

--------------------------------

### Run Import via CLI

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/getting-started/first-importer.md

Execute the import using the Artisan command, specifying the importer slug and the path to the data file.

```bash
php artisan ingest:run user-importer --file=path/to/users.csv
```

--------------------------------

### Enrich Data with External API Calls

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/index.md

Use external APIs to enrich data during the import process. This example fetches city data based on a postal code.

```php
// Enrich data during import with external APIs
->mapAndTransform('postal_code', 'city', function($postalCode, $row) {
    $response = Http::get("https://api.postal.com/{$postalCode}");
    return $response->json('city') ?? 'Unknown';
})
```

--------------------------------

### Configure Filesystem Disk

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

Ensure your filesystem disk is correctly configured in `config/filesystems.php`. This snippet shows a typical 'local' disk configuration.

```php
// config/filesystems.php
'disks' => [
    'local' => [
        'driver' => 'local',
        'root' => storage_path('app'),
    ],
],
```

--------------------------------

### Get Ingest Run Error Summary

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/api-reference.md

Retrieves an aggregated summary of errors for a specific ingest run by its ID. Useful for quick error analysis. Requires an Authorization token.

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  https://myapp.com/api/v1/ingest/123/errors/summary
```

--------------------------------

### SourceHandler Interface Definition

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/custom-source-handlers.md

Defines the contract for all custom source handlers, including methods for reading data, getting total rows, specifying processed file paths, and cleanup.

```php
interface SourceHandler
{
    /**
     * Read data from the source and yield rows as arrays.
     * @param mixed|null $payload Data from the trigger (e.g., UploadedFile, path, or custom data)
     */
    public function read(IngestConfig $config, mixed $payload = null): Generator;

    /**
     * Return the total number of rows, or null if unknown.
     */
    public function getTotalRows(): ?int;

    /**
     * Return the path where the processed file was stored, or null.
     */
    public function getProcessedFilePath(): ?string;

    /**
     * Clean up any temporary files or resources.
     */
    public function cleanup(): void;
}
```

--------------------------------

### Run Import via API

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/getting-started/first-importer.md

Trigger the import process by sending a multipart/form-data POST request to the auto-generated API endpoint, including the CSV file.

```bash
curl -X POST \
  -H "Authorization: Bearer <your-api-token>" \
  -F "file=@/path/to/users.csv" \
  https://your-app.com/api/v1/ingest/upload/user-importer
```

--------------------------------

### Enable Full Import Tracing

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Activate `withTracing()` to enable detailed logging for both mappings and transformations, which is beneficial for debugging complex imports.

```php
->withTracing()
```

--------------------------------

### Monitor Import Performance

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/index.md

Track import speed by calculating rows per second in the `afterRow` callback, logging the metric periodically to monitor performance.

```php
// Monitor import performance
->beforeRow(function(array &$row) {
    if (!isset($GLOBALS['import_start_time'])) {
        $GLOBALS['import_start_time'] = microtime(true);
        $GLOBALS['import_row_count'] = 0;
    }
    $GLOBALS['import_row_count']++;
})

->afterRow(function($model, array $row) {
    if ($GLOBALS['import_row_count'] % 1000 === 0) {
        $elapsed = microtime(true) - $GLOBALS['import_start_time'];
        $rows_per_sec = $GLOBALS['import_row_count'] / $elapsed;
        Log::info("Import speed: {$rows_per_sec} rows/sec");
    }
})
```

--------------------------------

### Configure Filesystem Source Type (Dynamic Path)

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

Use SourceType::FILESYSTEM for imports triggered by console commands or scheduled jobs when the file path is dynamic. The path is passed as a payload to Ingest::start() or via CLI argument.

```php
// IngestConfig for a command-triggered import where path is dynamic
->fromSource(SourceType::FILESYSTEM)
```

--------------------------------

### Handle Upload Errors with Screen Reader Announcement

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/ui-integration.md

Integrate the announceError function into your error handlers to provide accessible feedback for upload failures. This example updates a UI element and then announces the error.

```javascript
const handleUploadError = (error) => {
    errorMessage.value = 'Upload failed. Please try again.';
    announceError(errorMessage.value);
};
```

--------------------------------

### Increase Web Server Timeout Settings

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

For large uploads, increase timeout settings in your web server configuration (e.g., Nginx) to prevent connection timeouts. This example shows Nginx settings.

```nginx
client_max_body_size 100M;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
```

--------------------------------

### Using Synthetic Keys in Ingest Configuration

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Demonstrates how to use synthetic keys by combining fields at runtime using `beforeRow()` and then specifying the combined field with `keyedBy()`. The synthetic column must be a fillable attribute or database column.

```php
IngestConfig::for(Product::class)
    ->map('store_id', 'store_id')
    ->map('sku', 'sku')
    ->map('name', 'name')
    ->beforeRow(function (array &
$row) {
        $row['composite_key'] = $row['store_id'] . '|' . $row['sku'];
    })
    ->keyedBy('composite_key')
    ->onDuplicate(DuplicateStrategy::UPDATE);
```

--------------------------------

### Enable Transformation Tracing

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Use `traceTransformations()` to log how each value is transformed during the import process, focusing specifically on transformation steps.

```php
->traceTransformations()
```

--------------------------------

### Implement Permission-Based Import Access Control

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/security.md

Enforce authorization checks to ensure only users with the necessary permissions can perform imports. This example demonstrates using Laravel's gates and custom role checks.

```php
// In your ImportController
public function store(Request $request)
{
    $this->authorize('import', Product::class);
    
    // Additional role-based checks
    if (!auth()->user()->hasRole('data_manager')) {
        abort(403, 'Insufficient permissions for data import');
    }
    
    // Proceed with import
}
```

--------------------------------

### Vue.js Import Component with Axios

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/ui-integration.md

A Vue 3 component demonstrating file upload, API interaction for starting an import, and polling for status updates using Axios. Handles progress calculation and completion/failure conditions.

```javascript
<script setup>
import { ref } from 'vue';
import axios from 'axios';

const file = ref(null);
const run = ref(null);
const progress = ref(0);
const isPolling = ref(false);

const upload = async () => {
    const formData = new FormData();
    formData.append('file', file.value.files[0]);

    // 1. Start the Import
    const { data } = await axios.post('/api/v1/ingest/upload/product-importer', formData);
    
    run.value = data.data;
    startPolling(run.value.id);
};

const startPolling = (id) => {
    isPolling.value = true;
    
    const interval = setInterval(async () => {
        // 2. Check Status
        const { data } = await axios.get(`/api/v1/ingest/${id}`);
        const currentRun = data.data;
        
        // Update local state
        run.value = currentRun;
        
        // Calculate Percentage
        if (currentRun.progress.total > 0) {
            progress.value = Math.round(
                (currentRun.progress.processed / currentRun.progress.total) * 100
            );
        }

        // 3. Stop Condition
        if (['completed', 'completed_with_errors', 'failed'].includes(currentRun.status)) {
            clearInterval(interval);
            isPolling.value = false;
        }
        
    }, 2000); // Poll every 2 seconds
};
</script>

<template>
    <div>
        <input type="file" @change="file = $event.target" />
        <button @click="upload" :disabled="isPolling">Start Import</button>

        <div v-if="run">
            <h3>Status: {{ run.status }}</h3>
            
            <!-- Progress Bar -->
            <div style="background: #eee; height: 20px; width: 100%;">
                <div :style="{ width: progress + '%', background: 'blue', height: '100%' }"></div>
            </div>
            
            <p>
                Processed: {{ run.progress.processed }} / {{ run.progress.total }}
            </p>
            
            <div v-if="run.status === 'completed_with_errors'">
                ⚠️ Finished with {{ run.progress.failed }} errors.
            </div>
        </div>
    </div>
</template>
```

--------------------------------

### Define Reusable Product Mapping Class

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Create a `ProductMapping` class implementing `MappingInterface` and `NestedMappingInterface` to define reusable field mappings for products.

```php
use LaravelIngest\Contracts\HasMappings;
use LaravelIngest\Contracts\MappingInterface;
use LaravelIngest\Contracts\NestedMappingInterface;
use LaravelIngest\IngestConfig;
use LaravelIngest\NestedIngestConfig;
use LaravelIngest\Transformers\NumericTransformer;

class ProductMapping implements MappingInterface, NestedMappingInterface
{
    public function apply(IngestConfig $config, string $prefix = ''): IngestConfig
    {
        return $this->applyMappings($config, $prefix);
    }

    public function applyNested(NestedIngestConfig $config, string $prefix = ''): NestedIngestConfig
    {
        return $this->applyMappings($config, $prefix);
    }

    private function applyMappings(HasMappings $config, string $prefix = ''): HasMappings
    {
        $prefix = $prefix !== '' ? "{$prefix}_" : '';

        return $config
            ->map("{$prefix}product_id", 'product_id')
            ->map("{$prefix}product_name", 'name')
            ->mapAndTransform(
                "{$prefix}price_cents",
                'price',
                new NumericTransformer(decimals: 2)
            )
            ->map("{$prefix}sku", 'sku');
    }
}
```

--------------------------------

### Apply Reusable Mapping to Refund Importer

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Apply a `ProductMapping` directly without a prefix in the `RefundImporter` configuration to use the reusable product field mappings.

```php
use App\Models\Refund;
use LaravelIngest\Enums\SourceType;
use App\Ingest\Mappings\ProductMapping;

class RefundImporter implements IngestDefinition
{
    public function getConfig(): IngestConfig
    {
        return IngestConfig::for(Refund::class)
            ->fromSource(SourceType::UPLOAD)
            ->map('refund_id', 'id')
            ->applyMapping(new ProductMapping());  // No prefix needed
    }
}
```

--------------------------------

### Check File Existence with Storage Facade

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/advanced/troubleshooting.md

Before processing, verify if a file exists on a configured disk using the `Storage` facade. This helps diagnose 'File not found' issues.

```php
use Illuminate\Support\Facades\Storage;
Storage::disk('local')->exists('imports/file.csv'); // true/false
```

--------------------------------

### Define Importer Configuration

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/getting-started/first-importer.md

Manually create an importer class to define the import configuration, including source, keying, duplicate strategy, mapping, and validation.

```php
// app/Ingest/UserImporter.php
namespace App\Ingest;

use App\Models\User;
use LaravelIngest\Contracts\IngestDefinition;
use LaravelIngest\IngestConfig;
use LaravelIngest\Enums\SourceType;
use LaravelIngest\Enums\DuplicateStrategy;

class UserImporter implements IngestDefinition
{
    public function getConfig(): IngestConfig
    {
        return IngestConfig::for(User::class)
            ->fromSource(SourceType::UPLOAD)
            ->keyedBy('email')
            ->onDuplicate(DuplicateStrategy::UPDATE)
            ->map('full_name', 'name')
            ->map('user_email', 'email')
            ->mapAndTransform('is_admin', 'is_admin', fn($value) => $value === 'yes')
            ->validate([
                'full_name' => 'required|string',
                'user_email' => 'required|email'
            ]);
    }
}
```

--------------------------------

### Apply Reusable Mapping to Order Importer

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/ingest-config.md

Use `applyMapping()` with a `ProductMapping` instance and a prefix to include reusable product field mappings in the `OrderImporter` configuration.

```php
use App\Models\Order;
use LaravelIngest\Enums\SourceType;
use App\Ingest\Mappings\ProductMapping;

class OrderImporter implements IngestDefinition
{
    public function getConfig(): IngestConfig
    {
        return IngestConfig::for(Order::class)
            ->fromSource(SourceType::UPLOAD)
            ->map('order_id', 'id')
            ->map('customer_email', 'customer_email')
            ->applyMapping(new ProductMapping(), 'line_item');  // Prefix: line_item_product_id
    }
}
```

--------------------------------

### Get Importer Definition by Slug

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/usage/facade.md

Retrieve a specific importer's definition using its registered slug. This allows access to the importer's configuration, including model class, source type, chunk size, mappings, and validation rules.

```php
use LaravelIngest\Facades\Ingest;

// Get the importer definition
$definition = Ingest::getDefinition('product-importer');

// Access the configuration
$config = $definition->getConfig();

echo "Model: " . $config->modelClass;
echo "Source: " . $config->sourceType->value;
echo "Chunk size: " . $config->chunkSize;

// Check importer details
$mappings = $config->getMappings();
$validationRules = $config->getValidationRules();
```

--------------------------------

### Configure FTP Source Type

Source: https://github.com/zappzerapp/laravel-ingest/blob/main/docs/configuration/source-types.md

Use SourceType::FTP with the 'disk' and 'path' options to import files from a remote FTP server. The 'disk' must be configured in config/filesystems.php.

```php
// IngestConfig
->fromSource(SourceType::FTP, [
    'disk' => 'erp_ftp',
    'path' => '/exports/stock-levels.csv',
])
```
Tokens:
Raw