### EZS Script Configuration Example (INI)

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This INI file demonstrates a generic EZS script configuration for processing data. It includes environment setup, plugin usage, and asynchronous processing steps using delegates and forks. The script calls an external Python program for specific computation.

```ini
; fichier v1/lda.ini
[env]
; Cette variable d'environnement est définie dans le flux et tous ses sous-flux
path = generator
value = lda

[use]
; Les plugins utilisés sont disponibles aussi dans les sous-flux
; (même s'il est préférable de les redéclarer, au cas où un autre script voulait 
; réutiliser le même sous-script)
plugin = basics
plugin = analytics
plugin = spawn

# Step 1 (générique): Charger le fichier corpus
[delegate]
; delegate est une instruction qui ne fait qu'appeler un sous-flux sur tous les
; éléments du flux.
file = charger.cfg

# Step 2 (générique): Traiter de manière asynchrone les items reçus
[fork]
standalone = true
logger = logger.cfg

# Step 2.1 (spécifique): Lancer un calcul sur tous les items reçus
[fork/exec]
# command should be executable !
command = ./v1/lda.py
; Ici, le programme principal est écrit en Python.
; L'avantage c'est qu'il n'a pas à gérer le flux (problème complexe dans la 
; plupart des langages de programmation)
args = fix('-p')
args = env('nbTopic',6)

# Step 2.2 (générique): Enregistrer le résultat et signaler que le traitement est fini
[fork/delegate]
file = recorder.cfg

# Step 3 : Renvoyer immédiatement un seul élément indiquant comment récupérer le résultat
#          quand il sera prêt
[delegate]
file = recipient.cfg

```

--------------------------------

### Input for OBJFlatten Reverse Example

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This is the input JSON data for the '[OBJFlatten]' reverse example, demonstrating a flat structure with a dot-separated key ('racine.branche'). This input is processed by the ezs script to reconstruct the nested object.

```json
[
  {
    "racine.branche": "feuille",
    "logiciel": "lodex",
    "version": 15
  }
]
```

--------------------------------

### Générer identifiant de corpus avec singleton, env et get

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

Ce script génère un identifiant de corpus basé sur le champ 'département' du premier élément du flux. Il utilise 'singleton' pour capturer la valeur du premier élément, 'env' pour stocker cette valeur dans une variable d'environnement, et 'get' pour la récupérer et l'assigner au champ 'corpus' de tous les éléments.

```ini
[unpack]

[singleton]
[singleton/env]
path = corpus
value = get("département")

[assign]
path = corpus
value = env("corpus")

[pack]
```

--------------------------------

### Output of OBJFlatten Reverse Example

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This is the output JSON data after processing the input with the ezs script using '[OBJFlatten]' in reverse mode. It shows the reconstructed nested object, where 'racine.branche' has been transformed back into a nested structure: {"racine": {"branche": "feuille"}}.

```json
[
    {
        "racine": {
            "branche": "feuille"
        },
        "logiciel": "lodex",
        "version": 15
    }
]
```

--------------------------------

### Flatten JSON with Path Notation using OBJFlatten

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This snippet shows how to use the '[OBJFlatten]' instruction to transform nested JSON into a flat structure where nested keys become part of the column names (e.g., 'racine.branche'). This is often a precursor to flattening, as demonstrated in the example.

```json
[
    {
        "racine/branche": "feuille",
        "logiciel": "lodex",
        "version": 15
    }
]
```

--------------------------------

### Assign static or computed fields using assign

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Adds new fields or modifies existing ones in data objects using the 'assign' instruction. Values can be static strings or computed dynamically using expressions like 'get().multiply()'.

```ini
[unpack]

[assign]
path = double
value = get("id").multiply(2)

path = status
value = active

[dump]
indent = true

```

--------------------------------

### JSONL: Calculate Sum of Array Elements using Lodash reduce()

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This example demonstrates how to calculate the sum of elements in a JSON array using Lodash's 'reduce' function within an EZS script. It unpacks JSON lines, performs the sum calculation, and then packs the result.

```jsonl
{ "id":1, "val": [1,2,3]}
{ "id":2, "val": [3,4,5]}
```

--------------------------------

### Assign Dynamic Value Using Lodash Functions

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

Utilisez 'assign' avec des fonctions Lodash pour définir dynamiquement la valeur d'un champ. Le paramètre 'value' peut contenir des expressions Lodash qui opèrent sur les champs existants de l'objet, comme 'get("id").multiply(2)' pour doubler la valeur de l'identifiant.

```ini
[unpack]

[assign]
path = double
value = get("id").multiply(2)

[dump]
indent = true
```

--------------------------------

### Étendre numéros de département avec env, expand, update

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

Ce script utilise l'instruction 'env' pour stocker un objet de correspondance département-nom, 'expand' pour extraire le numéro de département, et 'update' pour remplacer le numéro par le nom correspondant en utilisant une fonction Lodash. Il gère les entrées JSONL et produit des sorties JSON.

```ini
[env]
path = noms
value = fix({ "54": "Meurthe-et-Moselle", "55": "Meuse", "57": "Moselle", "88": "Vosges" })

[unpack]

[expand]
size = 2
path = département

[expand/assign]
path = value
value = update("value", dept => env("noms")[dept]).get("value")

[pack]
```

--------------------------------

### Étendre numéros de département avec env, expand, thru

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

Alternative au script précédent, celui-ci utilise 'thru' de Lodash à la place de 'update' pour transformer le numéro de département en son nom correspondant, en conjonction avec 'env' et 'expand'.

```ini
[env]
path = noms
value = fix({ "54": "Meurthe-et-Moselle", "55": "Meuse", "57": "Moselle", "88": "Vosges" })

[unpack]

[expand]
size = 2
path = département

[expand/assign]
path = value
value = get("value").thru(dept => env("noms")[dept])

[pack]
```

--------------------------------

### Process JSONL with unpack/pack

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Demonstrates the use of 'unpack' and 'pack' instructions for processing data in JSON Lines format. This is a common pattern for handling streaming JSON data efficiently.

```ini
[unpack]
[pack]

```

--------------------------------

### Générer identifiant de corpus avec env et fix(Date.now())

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

Ce script attribue un identifiant de corpus unique à tous les éléments d'un flux en utilisant le timestamp actuel. L'instruction 'env' garantit que le timestamp est calculé une seule fois au début du traitement du flux, évitant ainsi des identifiants incohérents.

```ini
[env]
path = corpus
value = fix(Date.now())

[unpack]

[assign]
path = corpus
value = env("corpus")

[pack]
```

--------------------------------

### Send POST requests with URLConnect

Source: https://context7.com/parmentf/formation-ezs/llms.txt

This snippet demonstrates how to send POST requests to a web service for data enrichment using the URLConnect plugin. It specifies the target URL and expects JSONL input and output. Dependencies include the 'basics' plugin.

```ini
[use]
# URLConnect
plugin = basics

[unpack]

[URLConnect]
url = https://terms-extraction.services.istex.fr/v1/teeft/fr

[pack]
```

--------------------------------

### Process first element with singleton

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Applies instructions only to the first element of a data stream using the 'singleton' instruction. This is useful for operations that depend on initial stream conditions.

```ini
[unpack]

[singleton]
[singleton/env]
path = firstDept
value = get("département")

[assign]
path = corpus
value = env("firstDept")

[pack]

```

--------------------------------

### Parse JSON to JSONL using @ezs/basics

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Parses JSON input and converts it to JSONL format using the @ezs/basics plugin. This is useful for standardizing data into a line-delimited JSON structure.

```ini
[use]
# JSONParse
plugin = @ezs/basics

[JSONParse]

[dump]
indent = true

```

--------------------------------

### Lodex JSON Lines Loader with metadata

Source: https://context7.com/parmentf/formation-ezs/llms.txt

This script provides a complete loader for Lodex using JSON Lines format. It includes steps for unpacking, identifying missing URIs, deduplicating, flattening objects to prevent MongoDB issues, and stamping imported data with metadata like date, parser, and filename. It utilizes plugins from '@ezs/basics'.

```ini
append = pack
label = json-lines

[use]
plugin = basics

[unpack]

# Ensure URI field exists
[swing]
test = pick(['URI', 'uri']).pickBy(_.identity).isEmpty()
[swing/identify]

# Remove duplicates
[dedupe]
ignore = true

# Fix MongoDB incompatibility
[OBJFlatten]
separator = fix('.')
reverse = true

# Add import metadata
[assign]
path = lodexStamp.importedDate
value = fix(new Date()).thru(d => d.toDateString())
path = lodexStamp.usedParser
value = env('parser')
path = lodexStamp.uploadedFilename
value = env('source')
path = uri
value = get('uri').trim()
```

--------------------------------

### Convert CSV to JSONL using basics plugin

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Converts CSV data with headers into structured JSON objects, outputting each as a line in JSONL format. It utilizes the 'basics' plugin for parsing and conversion.

```ini
[use]
# CSVParse
plugin = basics

[CSVParse]
[CSVObject]
[pack]

```

--------------------------------

### INI: Configure CORHAL API Fetching and URI Assignment

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This INI configuration snippet sets up fetching data from the CORHAL API, including retry and timeout parameters. It then assigns a unique URI to each record by extracting it from the 'business.sourceUidChain' field.

```ini
[TXTConcat]

[replace]
path = q
value = self().trim()

[CORHALFetch]
url = https://corhal-api.inist.fr
retries = 3
timeout = 60000

[assign]
path = uri
value = get('business.sourceUidChain')
```

--------------------------------

### Convert JSONL to TSV using basics plugin

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Transforms JSONL data into Tab-Separated Values (TSV) format. It uses the 'basics' plugin, specifically the 'CSVString' instruction, to define the output format with a tab separator.

```ini
[use]
# CSVString
plugin = basics

[unpack]

[CSVString]
separator = fix("\t")

```

--------------------------------

### Lodex JSON Lines Loader Configuration (INI)

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This INI file configures a Lodex loader for JSON Lines format. It defines how to append data, sets a label, loads necessary plugins, unpacks data, ensures unique URIs, ignores duplicates, and flattens object keys to prevent issues with MongoDB.

```ini
append = pack
label = json-lines

# load some plugins to activate some statements
[use]
plugin = basics

[unpack]

# Ensures that each object contains an identification key (required by lodex)
[swing]
test = pick(['URI', 'uri']).pickBy(_.identity).isEmpty()
[swing/identify]

# Ignore objects with duplicate URI
[dedupe]
ignore = true

# Prevent keys form containing dot path notation (which is forbidden by nodejs mongoDB driver)
[OBJFlatten]
separator = fix('.')
reverse = true

# Uncomment to see each data sent to the database
#[debug]

```

--------------------------------

### Reconstruct JSON with OBJFlatten in ezs Script

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This ezs script utilizes '[JSONParse]', '[OBJFlatten]', and '[dump]' to parse input JSON, reconstruct a nested structure from dot-notation keys, and then pretty-print the output. The '[OBJFlatten]' instruction is configured with 'reverse = true' to achieve the desired structural transformation, making data compatible with MongoDB.

```ini
[JSONParse]

[OBJFlatten]
separator = fix(".")
reverse = true
safe = true

[dump]
indent = true

```

--------------------------------

### Assign Static Value to Object Field

Source: https://github.com/parmentf/formation-ezs/blob/master/core.md

L'instruction 'assign' permet de modifier ou d'ajouter un champ à un objet. Elle prend un 'path' pour désigner le champ et une 'value' pour la nouvelle valeur. La valeur peut écraser une valeur existante. Les paramètres 'path' et 'value' peuvent être répétés pour affecter plusieurs champs.

```ini
[unpack]

[assign]
path = un
value = 1

[dump]
indent = true
```

```ini
[assign]
path = un
path = deux

value = 1
value = 2
```

```ini
[assign]
path = un
value = 1

path = deux
value = 2
```

--------------------------------

### Use environment variables with env

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Stores and retrieves values using environment variables within the processing pipeline via the 'env' instruction. This allows for globally accessible state, like timestamps.

```ini
[env]
path = timestamp
value = fix(Date.now())

[unpack]

[assign]
path = corpus
value = env("timestamp")

[pack]

```

--------------------------------

### Batch process fields with expand

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Processes field values in batches using the 'expand' instruction, which applies a sub-pipeline to groups of records. This is useful for operations requiring context from multiple items.

```ini
[unpack]

[expand]
size = 2
path = valeur

[expand/assign]
path = value
value = get("value").multiply(2)

[pack]

```

--------------------------------

### INI: EZS Script to Calculate Sum of Array Elements

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This INI script, used with JSONL input, unpacks each line, calculates the sum of the 'val' array using Lodash's 'reduce' function, assigns the sum to a 'somme' field, and then repacks the enhanced JSON line.

```ini
[unpack]

[assign]
path = somme
value = get("val").reduce((somme, valeur) => somme + valeur, 0)

[pack]
```

--------------------------------

### JSONL: Output after Calculating Sum of Array Elements

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This is the resulting JSONL output after applying the EZS script to calculate the sum of the 'val' array for each JSON object. The 'somme' field now contains the calculated sum.

```jsonl
{"id":1,"val":[1,2,3],"somme":6}
{"id":2,"val":[3,4,5],"somme":12}
```

--------------------------------

### Rate limit processing with throttle

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Controls the processing speed of the stream using the 'throttle' instruction, limiting operations per unit of time. This is crucial for managing API call rates or resource consumption.

```ini
[unpack]

[throttle]
bySecond = 1

[assign]
path = time
value = fix(String(new Date()).slice(16,24))

[pack]

```

--------------------------------

### Reconstruct JSON Structure with OBJFlatten (Reverse Mode)

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This snippet illustrates the use of '[OBJFlatten]' with 'reverse = true' to reconstruct a hierarchical JSON structure from a flat representation. This is particularly useful for ensuring compatibility with databases like MongoDB that do not support dot notation in field names. It takes a flat JSON object with path-like keys and outputs a nested JSON object.

```ini
# Prevent keys from containing dot path notation (which is forbidden by nodejs mongoDB driver)
[OBJFlatten]
separator = fix('.')
reverse = true
safe = true

```

--------------------------------

### Parse French CSV with custom delimiters and quotes

Source: https://context7.com/parmentf/formation-ezs/llms.txt

This snippet demonstrates advanced CSV parsing for French format, allowing custom delimiters (semicolon) and quote characters. It uses the CSVParse, CSVObject, and CSVString plugins. The input is CSV, and the output is TSV.

```ini
[use]
# CSVParse CSVObject CSVString
plugin = basics

[CSVParse]
separator = ;
quote = fix('"')

[CSVObject]

[CSVString]
separator = fix("\t")
```

--------------------------------

### Prevent MongoDB dot notation issues with OBJFlatten

Source: https://context7.com/parmentf/formation-ezs/llms.txt

This snippet uses the OBJFlatten plugin to reverse-flatten objects, preventing issues with MongoDB's dot notation for key names. It sets a custom separator and enables reverse and safe modes. The input is JSON, and the output is a JSON object with nested structures.

```ini
[use]
plugin = @ezs/basics

[JSONParse]

[OBJFlatten]
separator = fix(".")
reverse = true
safe = true

[dump]
indent = true
```

--------------------------------

### Flatten JSON to Tabular Data using objects2columns

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This snippet demonstrates how the 'objects2columns' instruction flattens nested JSON elements into a single-level tabular structure. It's useful for simplifying complex data for reporting purposes. The input is a nested JSON object, and the output is a flattened JSON array.

```json
[
  {
    "racine": {
      "branche": "feuille"
    },
    "logiciel": "lodex",
    "version": 15
  }
]
```

```json
[
  {
    "racine": "{\"branche\":\"feuille\"}",
    "logiciel": "lodex",
    "version": 15
  }
]
```

--------------------------------

### Deduplicate entries with dedupe

Source: https://context7.com/parmentf/formation-ezs/llms.txt

Removes duplicate records from the stream based on specified field values using the 'dedupe' instruction. The 'ignore' option can be used to control behavior with repeated values.

```ini
[unpack]

[dedupe]
path = département
ignore = true

[pack]

```

--------------------------------

### INI: Exclude Fields using Lodash omit() in Exchange

Source: https://github.com/parmentf/formation-ezs/blob/master/advanced.md

This INI configuration uses the '[exchange]' instruction with Lodash's 'omit()' function to remove specified fields from a record. This is useful for cleaning up data and keeping only necessary fields.

```ini
[exchange]
value = omit(["abstract","authors","classifications","business","pii","arxiv","inspire","localRef","pmcId","articleNumber","nnt","ppn","origins","technical","halId","title","originalGenre","pmId","fulltextUrl","enrichments","host"])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.