### Install @ezs/istex Package Source: https://github.com/inist-cnrs/ezs/blob/master/packages/istex/README.md Installs the @ezs/istex package using npm. This is the first step to use the ISTEX plugin in your project. It requires Node.js and npm to be installed. ```bash npm install @ezs/istex ``` -------------------------------- ### Example documentation script adaptation Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Example of adapting a 'documentation' script to generate READMEs within a monorepo structure. ```bash "doc": "documentation readme src/* --markdown-toc-max-depth=2 --readme-file=../../docs/plugin-sparql.md --section=usage" ``` -------------------------------- ### Install @ezs/basics Package Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md This command installs the @ezs/basics package using npm. It is a prerequisite for using the data transformation functionalities provided by the library. ```bash npm install @ezs/basics ``` -------------------------------- ### Install EZS SPARQL Plugin Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-sparql.md Installs the @ezs/sparql npm package using npm. This is the first step to use the SPARQL functionalities provided by the plugin. ```bash npm install @ezs/sparql ``` -------------------------------- ### Install Dependencies with Yarn Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md Installs all project dependencies using Yarn. This is a prerequisite for building and running the project. ```bash yarn install ``` -------------------------------- ### Install @ezs/core and @ezs/strings Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-strings.md Installs the necessary @ezs/core and @ezs/strings packages using npm. These are prerequisites for using the string manipulation functionalities. ```bash npm install @ezs/core npm install @ezs/strings ``` -------------------------------- ### affAlign input and output example Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Shows an example of the input JSON structure for the affAlign script and the corresponding output after processing, where 'conditorRnsr' is added to the affiliations. ```json [{ "xPublicationDate": ["2012-01-01", "2012-01-01"], "authors": [{ "affiliations": [{ "address": "GDR 2989 Université Versailles Saint-Quentin-en-Yvelines, 63009" }] }] }] ``` ```json [{ "xPublicationDate": ["2012-01-01", "2012-01-01"], "authors": [{ "affiliations": [{ "address": "GDR 2989 Université Versailles Saint-Quentin-en-Yvelines, 63009", "conditorRnsr": ["200619958X"] }] }] }] ``` -------------------------------- ### JSON Input Example 1 Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Provides an example of JSON input data, consisting of a list of objects with 'id' and 'value' properties, intended for processing by the ezs project. ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2003, "value": 3 }, { "id": 2005, "value": 4 }, { "id": 2007, "value": 5 }, { "id": 2009, "value": 6 }, { "id": 2011, "value": 7 }, { "id": 2013, "value": 8 } ] ``` -------------------------------- ### Install Conditor Plugin with npm Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-conditor.md Installs the necessary packages for the Conditor plugin and the EZS core. Requires npm to be installed. ```bash npm install @ezs/core npm install @ezs/conditor ``` -------------------------------- ### Add a feature commit example Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Example of a conventional commit message for adding a new feature to a specific package. ```bash git commit -m "feat (foo): Add marvelous new feature bar" ``` -------------------------------- ### Conventional Commit Example (feat) Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md An example of a Git commit following the Conventional Commits specification for a new feature in the 'foo' package. This format aids in automated changelog generation. ```bash git commit -m "feat(foo): Add a new feature to the foo package" ``` -------------------------------- ### Clone an old package Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Example command to clone an existing Git repository for an old package. ```bash git clone https://github.com/touv/node-ezs ../old-ezs/core ``` -------------------------------- ### JSON Input Example for Value Function Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md These JSON examples represent two distinct input datasets. Dataset 1 contains simple key-value pairs, while Dataset 2 features nested objects. Both are used to demonstrate the 'value' function's behavior. ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2003, "value": 3 }, { "id": 2005, "value": 4 }, { "id": 2007, "value": 5 }, { "id": 2009, "value": 6 }, { "id": 2011, "value": 7 }, { "id": 2013, "value": 8 } ] ``` ```json [ { "id": 1, "value": { "hello": "world" } }, { "id": 2, "value": { "hello": "ezs" } }, { "id": 3, "value": { "hello": "lodex" } } ] ``` -------------------------------- ### Lerna Bootstrap Packages Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md Bootsraps all packages in the Lerna monorepo, installing dependencies and linking packages. This command ensures the monorepo is in a consistent state. ```bash npx lerna bootstrap ``` -------------------------------- ### Example README badges for scoped package Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Markdown snippet showing how to add npm badges to a README file for a scoped package like '@ezs/sparql'. ```markdown [![npm version](https://img.shields.io/npm/v/@ezs/sparql)](https://npm.im/@ezs/sparql) [![license](https://img.shields.io/npm/l/@ezs/sparql)](https://npm.im/@ezs/sparql) ``` -------------------------------- ### Example Babel build script adaptation Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Shows how to adapt a Babel build script for a monorepo context using '--root-mode upward'. ```json "build": "babel --root-mode upward src --out-dir lib" ``` -------------------------------- ### Fetch Data from URLs/APIs with EZS Source: https://context7.com/inist-cnrs/ezs/llms.txt This example demonstrates fetching data from URLs and APIs using EZS, including automatic retries, timeout handling, and JSON parsing. Requires the 'basics' plugin. ```ini [use] plugin = basics ``` -------------------------------- ### Breaking change commit example Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Example of a conventional commit message for introducing a breaking change, including a BREAKING_CHANGE description. ```bash git commit -m 'fix (bar): Fixed issue #4 BREAKING_CHANGE: baz function has changed parameters ' ``` -------------------------------- ### Create Tar Archives with TARDump (Parameters) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md Builds a tar archive from received objects. It allows specifying manifest location, output location, JSON conversion for contents, file extension, additional files, and gzip compression. The input is typically an empty object signifying the start of the operation, with parameters controlling the archive's structure and content. ```json { "manifest": "/path/to/manifest.json", "location": "output/directory", "json": true, "extension": "txt", "additionalFile": "/path/to/extra.file", "compress": true } ``` -------------------------------- ### Fix a bug commit example Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Example of a conventional commit message for fixing a bug in a specific package. ```bash git commit -m "fix (core): Chuck Norris got it!" ``` -------------------------------- ### Conventional Commit Example (fix) Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md An example of a Git commit following the Conventional Commits specification for a fix in the 'core' package. This format aids in automated changelog generation. ```bash git commit -m "fix(core): Fix a bug in the core package" ``` -------------------------------- ### compareRnsr input and output example Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Provides an example of the input JSON structure for the compareRnsr script, which includes 'rnsr' and 'conditorRnsr' fields in affiliations, and the resulting JSON output detailing the comparison metrics. ```javascript [{ "authors": [{ "affiliations": [{ "address": "GDR 2989 Université Versailles Saint-Quentin-en-Yvelines, 63009", "rnsr": ["200619958X"], "conditorRnsr": ["200619958X"] }] }] }] ``` ```javascript { "correct": 1, "total": 1, "recall": 1 } ``` -------------------------------- ### JSON Output Example Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md This JSON structure represents a sample output, showcasing a list of objects, each with a unique ID and a nested 'value' object containing 'id', 'value', and 'label'. ```json [ { "id": "0000000000000000001.00000000000000000000", "value": { "id": 1, "value": 1, "label": "static value" } }, { "id": "0000000000000000002.00000000000000000000", "value": { "id": 2, "value": 2, "label": "static value" } } ] ``` -------------------------------- ### JSON Input Example 2 Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Another JSON input example for the ezs project, containing a list of objects where the 'value' property is itself a nested JSON object. This demonstrates flexibility in data structure. ```json [ { "id": 1, "value": { "hello": "world" } }, { "id": 2, "value": { "hello": "ezs" } }, { "id": 3, "value": { "hello": "lodex" } } ] ``` -------------------------------- ### Process data with affAlign and compareRnsr scripts Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Demonstrates how to pipe data through the affAlign script for affiliation alignment and then through the compareRnsr script for comparison. Requires installation of '@ezs/basics' as well. ```bash $ ./bin/affAlign.js < data/1000-notices-conditor-hal.json | ./bin/compareRnsr.js recall: 0.7104885057471264 correct: 989 total: 1392 ``` -------------------------------- ### JSON Output Example for Value Function Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md These JSON outputs illustrate the results of applying the 'value' function to the provided input datasets. Dataset 1 output shows an array of numerical values, while Dataset 2 output displays an array of objects with extracted 'hello' properties. ```json [ 1, 2, 3, 4, 5, 6, 7, 8 ] ``` ```json [ { "hello": "world" }, { "hello": "ezs" }, { "hello": "lodex" } ] ``` -------------------------------- ### INI Script for Analytics Plugin Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md This INI script demonstrates how to import the analytics plugin and use the 'value' function with its default settings. It's a configuration example for a specific tool or framework. ```ini ; Import analytics plugin required to use "value" ; Importation du plugin analytique nécessaire pour utiliser "value" [use] plugin = analytics ; Using "value" with default settings ; Utilisation de "tune" avec les paramètres par défaut [value] ; path = value ``` -------------------------------- ### JSON Output Example 1 (Processed) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Shows the JSON output after processing the first input dataset using the 'value' function. The output is a simple array of the 'value' fields from the input. ```json [ 1, 2, 3, 4, 5, 6, 7, 8 ] ``` -------------------------------- ### Output / Sortie Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Retrieves a specified portion of the stream. Supports pagination with start and size parameters. ```APIDOC ## GET /output ### Description Retrieves a specified portion of the stream. Supports pagination with start and size parameters. ### Method GET ### Endpoint /output ### Parameters #### Query Parameters - **start** (Number) - Optional - The beginning index of the specified portion of the stream. Defaults to 1. - **size** (Number) - Optional - The size of the specified portion of the stream. Defaults to 10. ### Response #### Success Response (200) - **Array of Objects** - An array of objects, each with 'id' and 'value' properties. #### Response Example ```json [ { "id": 2023, "value": 12 }, { "id": 2021, "value": 11 }, { "id": 2019, "value": 10 }, { "id": 2017, "value": 9 }, { "id": 2013, "value": 8 }, { "id": 2011, "value": 7 }, { "id": 2009, "value": 6 }, { "id": 2007, "value": 5 }, { "id": 2005, "value": 4 }, { "id": 2003, "value": 3 } ] ``` ``` -------------------------------- ### Tune Identifiers Example Input (JSON) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md This JSON shows the input structure for the 'tune' function. It contains objects with 'id' and 'value' fields that will be processed to create unified identifiers for sorting. ```json Input / Entrée ```json [ { "id": 1, "value": 1 }, { "id": 2, "value": 2 } ] ``` ``` -------------------------------- ### JSON: Input for CORHALFetch Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example JSON input for the CORHALFetch function, specifying a query parameter. ```json [ { q: "toto" }, ] ``` -------------------------------- ### JSON Output Example 2 (Processed) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Illustrates the JSON output after processing the second input dataset with the 'value' function. The output retains the structure of the nested 'value' objects from the input. ```json [ { "hello": "world" }, { "hello": "ezs" }, { "hello": "lodex" } ] ``` -------------------------------- ### JSON: Input for OAFetch Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example JSON input for the OAFetch function, specifying a filter parameter for OpenAlex API queries. ```json [ { filter: "authorships.author.id:a5000387389" }, ] ``` -------------------------------- ### Tuning Functionality Input/Output (JSON) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md This section provides an example of the input and output for the 'tune' function. The input is a JSON array of objects, each containing an 'id' and a 'value'. The 'tune' function processes these objects to create a unified ID, preparing the data for sorting, although the output structure reflecting the unified ID is not explicitly shown here, only the input data. ```json // Input [ { "id": 1, "value": 1 }, { "id": 2, "value": 2 } ] ``` -------------------------------- ### Add modules to ezs using ezs.use Source: https://github.com/inist-cnrs/ezs/blob/master/docs/api.md The ezs.use function allows for the addition of statement bundles (modules) to the ezs core. This extends the library's capabilities. Ensure the module is installed via npm and imported before use. ```javascript import ezs from '@ezs/core'; import basics from '@ezs/basics'; import files from '@ezs/files'; ezs.use(basics); ezs.use(files); ``` -------------------------------- ### Delegate Sub-Pipelines in EZS Source: https://context7.com/inist-cnrs/ezs/llms.txt This example demonstrates sub-pipeline delegation in EZS for complex nested processing. It shows how to execute external pipelines for shared logic, item-specific transformations, array field processing, and conditional execution. Requires the 'basics' plugin. ```ini [use] plugin = basics [delegate] file = ./enrich.ini [spawn] file = ./transform.ini [map] path = items file = ./process-item.ini [swing] test = get('type').equals('document') file = ./process-document.ini ``` -------------------------------- ### JSON: Output from getRnsrInfo Function Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example JSON output from the getRnsrInfo function, providing comprehensive details about a specific RNSR entry. ```json [ { "an_fermeture": "", "annee_creation": "2014", "code_postal": "75015", "etabAssoc": [{ "etab": { "libelle": "Centre national de la recherche scientifique", "libelleAppauvri": "centre national de la recherche scientifique", "sigle": "CNRS", "sigleAppauvri": "cnrs" }, "label": "UMR", "labelAppauvri": "umr", "numero": "8253" }, { "etab": { "libelle": "Institut national de la sante et de la recherche medicale", "libelleAppauvri": "institut national de la sante et de la recherche medicale", "sigle": "INSERM", "sigleAppauvri": "inserm" }, "label": "U", "labelAppauvri": "u", "numero": "1151" }, { "etab": { "libelle": "Université Paris Cité", "libelleAppauvri": "universite paris cite", "sigle": "U PARIS Cité", "sigleAppauvri": "u paris cite" }, "label": "UM", "labelAppauvri": "um", "numero": "111" }], "intitule": "Institut Necker Enfants Malades - Centre de médecine moléculaire", "intituleAppauvri": "institut necker enfants malades centre de medecine moleculaire", "num_nat_struct": "201420755D", "sigle": "INEM", "sigleAppauvri": "inem", "ville_postale": "PARIS", "ville_postale_appauvrie": "paris" } ] ``` -------------------------------- ### Create Object Array from Values (TXTObject) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md TXTObject transforms an array of values into an array of objects, where each object has a specified key (defaulting to 'value') and the corresponding value from the input array. Example input: `[1, "b"]`, output: `[{ "value": 1 }, { "value": "b" }]`. ```javascript txtObject: (array, key = "value") => { return array.map(item => ({ [key]: item })); } ``` -------------------------------- ### Aggregate and Analyze Data with EZS Source: https://context7.com/inist-cnrs/ezs/llms.txt This example shows how to perform data aggregation and statistical analysis on streaming data using EZS. It parses CSV, counts occurrences of specific fields, aggregates identical values, sums counts, and sorts the results. Requires 'basics' and 'analytics' plugins. ```ini [use] plugin = basics plugin = analytics [CSVParse] [CSVObject] [count] path = city [aggregate] [summing] [sort] path = value reverse = true [dump] indent = true ``` -------------------------------- ### Connect to URL Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Establishes a connection to a given URL. This is a foundational instruction for network-related operations. ```javascript const url = 'http://example.com'; const connection = await URLConnect(url); // 'connection' can be used for further network operations ``` -------------------------------- ### Split String by Separator (TXTParse) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md TXTParse takes a string or an array of strings and splits them into individual elements based on a specified separator (defaulting to newline '\n'). It returns an array of the split strings. Example input: `["a\nb\n", "c\nd\n"]`, output: `["a", "b", "c", "d"]`. ```javascript txtParse: (array, separator = "\n") => { return array.reduce((acc, val) => { return acc.concat(val.split(separator).filter(Boolean)); }, []); } ``` -------------------------------- ### Lint JavaScript Files with Lerna Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Lints JavaScript files for a specific package within the EZS project using Lerna. This command targets a single package, identified by its scope, to run the linting process. Requires Lerna to be installed as a dev dependency. ```bash npx lerna run lint --scope=@ezs/analytics ``` -------------------------------- ### Concatenate String Array to Single String (TXTConcat) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md TXTConcat takes an array of strings and concatenates them into a single string. It has no specific parameters and returns the combined string. Example input: `["a", "b"]`, output: `["ab"]`. ```javascript txtConcat: (array) => { return array.join(""); } ``` -------------------------------- ### EZS Command Line Script Execution Source: https://context7.com/inist-cnrs/ezs/llms.txt Demonstrates how to execute EZS scripts directly from the command line. It covers basic execution, passing parameters, enabling debugging, using environment variables, and piping data between different EZS commands or other command-line tools. ```bash # Basic execution ezs script.ini < input.json > output.json # With parameters ezs -p size=100 -p format=csv script.ini < data.json # With debugging DEBUG=ezs:* ezs script.ini < input.json # Using environment variables API_KEY=secret ezs api-script.ini < requests.json # Pipe between commands cat data.csv | ezs csv-to-json.ini | ezs enrich.ini | ezs json-to-csv.ini > output.csv ``` -------------------------------- ### Configure strings plugin Source: https://github.com/inist-cnrs/ezs/blob/master/packages/strings/README.md Initializes the strings plugin at the beginning of a script by specifying 'plugin = strings' in the configuration. ```ini [use] plugin = strings ``` -------------------------------- ### Run all tests with npm Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Command to run all tests defined within the project using npm. ```bash npm test ``` -------------------------------- ### Build all packages with Lerna and Babel Source: https://github.com/inist-cnrs/ezs/blob/master/CONTRIBUTING.md Command to run the build script (likely transpilation with Babel) for all packages managed by Lerna. ```bash npx lerna run build ``` -------------------------------- ### JSON: Output from CORHALFetch Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example JSON output format from the CORHALFetch function, typically an array of objects. ```json [{...}, {"a": "b"}, {"a": "c" }] ``` -------------------------------- ### GET /v1/records Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/data/README.md Retrieve records from the Conditor Data API based on specified query parameters. ```APIDOC ## GET /v1/records ### Description Retrieve records from the Conditor Data API based on specified query parameters. This endpoint allows filtering and retrieving detailed information about records. ### Method GET ### Endpoint https://api-integ.conditor.fr/v1/records ### Parameters #### Query Parameters - **q** (string) - Required - The query string to filter records. Example: `"source:hal AND authors>affiliations>\"*\""` - **page_size** (integer) - Optional - The number of records to return per page. Default is 10. - **includes** (string) - Optional - Specifies additional fields to include in the response. Example: `authors,xPublicationDate` - **debug** (boolean) - Optional - Enables debug mode for the query. - **access_token** (string) - Required - Authentication token for accessing the API. ### Request Example ```bash curl \ 'https://api-integ.conditor.fr/v1/records?q=%22source%3Ahal%20AND%20authors%3Eaffiliations%3E%5C%22*%5C%22%22&page_size=10&includes=authors%2CxPublicationDate&access_token=ey...' \ > data/10-notices-conditor-hal.json ``` ### Response #### Success Response (200) - **records** (array) - An array of record objects. - **total** (integer) - The total number of records matching the query. #### Response Example ```json { "records": [ // ... record objects ... ], "total": 100 } ``` ``` -------------------------------- ### Delegate processing to an external pipeline (spawn) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-core.md The 'spawn' script delegates processing to an external pipeline for each chunk of data. It functions similarly to 'delegate' but uses a separate external pipeline for each chunk. Parameters include file paths, script strings, commands, logger, and an optional cache. ```json { "spawn": { "script": "another_pipeline_script" } } ``` -------------------------------- ### JSON: Output from getRnsr Function Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example JSON output from the getRnsr function, showing an ID and a value array containing RNSR identifiers. ```json [{ "id": 1, "value": ["200619958X"] }] ``` -------------------------------- ### Fetch Data from URL Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Retrieves data from a specified URL. This instruction is commonly used for downloading resources or making API requests. ```javascript const url = 'http://example.com/data.json'; const data = await URLFetch(url); // 'data' will contain the content fetched from the URL ``` -------------------------------- ### Build All Packages with Yarn Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md Builds all packages within the monorepo. This command compiles the code for all individual packages, preparing them for use or testing. ```bash yarn build ``` -------------------------------- ### Replace Namespaces with Prefixes in Object Keys - Ini/Configuration Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md This script configuration defines how to replace namespaces (full URIs) with shorter prefixes in object keys. It uses a basic plugin and specifies mapping rules for prefixes and their corresponding namespaces. The input is a JSON array of objects with URI keys, and the output is a JSON array with prefixed keys. ```ini [use] plugin = basics [OBJNamespaces] prefix = dc: namespace = http://purl.org/dc/terms/ prefix = place: namespace = http://purl.org/ontology/places# ``` -------------------------------- ### Make Request to URL Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Sends an HTTP request to a specified URL. This instruction provides flexibility in defining request methods, headers, and body. ```javascript // Example POST request const url = 'http://example.com/api'; const options = { method: 'POST', body: JSON.stringify({ data: 'payload' }) }; const response = await URLRequest(url, options); // 'response' will contain the server's response ``` -------------------------------- ### Configure Conditor token for conditorScroll Source: https://github.com/inist-cnrs/ezs/blob/master/packages/conditor/README.md Example of how to set the Conditor API token in a .env file for the conditorScroll function to use. This is a security measure to authenticate API requests. ```bash CONDITOR_TOKEN=eyJhbG..." ``` -------------------------------- ### FILELoad: Load file content by chunk Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md The FILELoad component reads content from files specified by filename. It supports optional gzip compression. It takes an array of filenames and outputs their content, potentially in chunks. The location parameter specifies the directory where files are stored. ```ini [use] plugin = analytics plugin = basics [FILELoad] location = /tmp [CSVParse] ``` -------------------------------- ### Configure WOSFetch with Token Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-conditor.md This snippet shows the configuration for WOSFetch, specifying the authentication token required to access the Web of Science API. It is typically used in a configuration file. ```ini [WOSFetch] token = SDQedaeaazedsqsd ``` -------------------------------- ### Fetch and Paginate GitHub Repositories with URLRequest, URLPagination, and URLStream Source: https://context7.com/inist-cnrs/ezs/llms.txt This configuration fetches repository data from the GitHub API. It includes settings for URL requests with retries and timeouts, pagination through results, and streaming the items. The `URLRequest` block defines basic fetch parameters, `URLPagination` handles total counts and page limits, and `URLStream` specifies the URL, item path, and timeout for streaming. Finally, a `dump` command is used to output the results. ```ini # Fetch from API with query parameters [URLRequest] url = https://api.github.com/search/repositories json = true timeout = 10000 retries = 3 noerror = false # Paginate through results [URLPagination] total = get('total_count') limit = 30 maxPages = 10 # Fetch each page [URLStream] url = https://api.github.com/search/repositories path = .items.* timeout = 10000 [dump] indent = true ``` -------------------------------- ### Lerna Create New Package Source: https://github.com/inist-cnrs/ezs/blob/master/AGENTS.md Creates a new package within the Lerna monorepo. This command initializes a new package with a default structure. ```bash npx lerna create @ezs/package ``` -------------------------------- ### URLParse API Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md The URLParse endpoint takes a URL string, parses it, and returns an object containing its components such as protocol, host, pathname, etc. It also supports converting such an object back to a URL string. ```APIDOC ## URLParse ### Description Take an URL `String`, parse it and return `Object`. Fields of the returned object: `href`, `origin`, `protocol`, `username`, `password`, `host`, `hostname`, `port`, `pathname`, `search`, `hash`. URLString statement convert such an object to a string. ### Method POST (Assumed based on description) ### Endpoint /ezs/urlparse (Assumed) ### Parameters #### Request Body - **url** (String) - The URL string to parse. ### Request Example ```json { "url": "https://example.com/path?query=string#hash" } ``` ### Response #### Success Response (200) - **href** (String) - The full URL. - **origin** (String) - The URL's origin (protocol + host). - **protocol** (String) - The URL's protocol. - **username** (String) - The URL's username. - **password** (String) - The URL's password. - **host** (String) - The URL's host (hostname + port). - **hostname** (String) - The URL's hostname. - **port** (String) - The URL's port. - **pathname** (String) - The URL's pathname. - **search** (String) - The URL's query string. - **hash** (String) - The URL's hash. #### Response Example ```json { "href": "https://example.com/path?query=string#hash", "origin": "https://example.com", "protocol": "https:", "username": "", "password": "", "host": "example.com", "hostname": "example.com", "port": "", "pathname": "/path", "search": "?query=string", "hash": "#hash" } ``` ``` -------------------------------- ### Filter Data Greater Than Value (JSON Input/Output) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Demonstrates filtering an array of objects based on a numeric field ('id' in this example) being greater than a specified value. The 'strict' parameter in the configuration determines if the comparison includes equality. ```json [ { id: 2000, value: 1 }, { id: 2001, value: 2 }, { id: 2003, value: 3 }, { id: 2005, value: 4 }, { id: 2007, value: 5 }, { id: 2009, value: 6 }, { id: 2011, value: 7 }, { id: 2013, value: 8 }, ] ``` ```json [ { id: 2005, value: 4 }, { id: 2007, value: 5 }, { id: 2009, value: 6 }, { id: 2011, value: 7 }, { id: 2013, value: 8 }, ] ``` -------------------------------- ### Handle URL Pagination Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Manages pagination for URLs, allowing for fetching data across multiple pages of a paginated resource. It automates the process of moving from one page to the next. ```javascript // Assuming 'baseUrl' and 'paginationParams' are defined const paginatedDataStream = await URLPagination(baseUrl, paginationParams); // 'paginatedDataStream' will emit data from all paginated results ``` -------------------------------- ### Create Tar Archive from Objects - JavaScript Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Builds a tar file from received objects. It allows specifying manifest and location paths, converting chunk content to JSON, defining file extensions, adding additional files, and enabling gzip compression. The input is typically an object with configuration parameters. ```javascript // Assuming a function TARDump exists and is imported/available // Example Usage: const tarConfig = { manifest: "path/to/manifest.json", // Optional location: "output/directory", // Optional: default "data" json: true, // Optional: default true extension: "json", // Optional: default "json" additionalFile: "path/to/extra.txt", // Optional compress: true // Optional: default false }; // The TARDump function would process data based on tarConfig. // TARDump(tarConfig); ``` -------------------------------- ### Distribute Object Values Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-analytics.md Distributes values from an input object into a series of numbers. It takes objects with 'id' and 'value' and generates a sequence based on parameters like 'step', 'start', and 'size'. Missing values are defaulted to 0. ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2003, "value": 3 }, { "id": 2005, "value": 4 }, { "id": 2007, "value": 5 }, { "id": 2009, "value": 6 }, { "id": 2011, "value": 7 }, { "id": 2013, "value": 8 }, ] ``` ```ini [use] plugin = analytics [distribute] ``` ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2002, "value": 0 }, { "id": 2003, "value": 3 }, { "id": 2004, "value": 0 }, { "id": 2005, "value": 4 }, { "id": 2006, "value": 0 }, { "id": 2007, "value": 5 }, { "id": 2008, "value": 0 }, { "id": 2009, "value": 6 }, { "id": 2010, "value": 0 }, { "id": 2011, "value": 7 }, { "id": 2012, "value": 0 }, { "id": 2013, "value": 8 } ] ``` -------------------------------- ### Load multiple plugins using 'use' in INI format Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-core.md Loads one or more plugins for use in the data processing pipeline. The 'use' directive specifies the plugin names to be loaded. Multiple 'plugin' entries can be used to load several plugins. ```ini [use] plugin = basics plugin = analytics ``` -------------------------------- ### FILEMerge: Merge data from multiple files Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md FILEMerge combines data from multiple sources (objects or buffers) into a single document. It's useful for consolidating information from various inputs. The script example shows its integration with FILELoad and a replace step. ```ini [use] plugin = basics [FILELoad] [FILEMerge] [replace] path = contentOfFile1AndFile2 value = self() ``` -------------------------------- ### Get File Contents using JavaScript Source: https://github.com/inist-cnrs/ezs/blob/master/packages/teeft/README.md Retrieves the content of specified files. Takes an array of file paths as input and returns an array of objects, each containing the file path and its content. Useful for batch processing file data. ```javascript /* * Takes an array of file paths and returns their content. * Returns: Array of { path: string, content: string }. */ const fileContents = await TeeftGetFilesContent(['/path/to/file1.txt', '/path/to/file2.txt']); ``` -------------------------------- ### Use plugins with EZS Source: https://github.com/inist-cnrs/ezs/blob/master/packages/core/README.md The 'use' function in EZS is primarily used to load plugins. Multiple 'plugin' directives can be specified to load several plugins at once. This allows for extending EZS functionality with custom or third-party modules. ```ini [use] plugin = basics plugin = analytics ``` -------------------------------- ### Delegate processing to an external pipeline (singleton) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-core.md The 'singleton' script delegates processing to an external pipeline. This can be defined via a file, a script string, commands object, or a URL-like command. It also supports a logger for error handling. ```json { "singleton": { "file": "path/to/external/pipeline.ezs" } } ``` -------------------------------- ### Get ISTEX Facets with ISTEXFacet Source: https://github.com/inist-cnrs/ezs/blob/master/packages/istex/README.md Retrieves aggregations from the ISTEX API based on a query string and a specified facet. It accepts optional parameters for the query, facet name, and a user-agent identifier (sid). The result is an array of facet aggregation objects. ```javascript from([{ query: 'ezs', facet: 'corpusName' }]) .pipe(ezs('ISTEXFacet', { sid: 'test', })) ``` -------------------------------- ### TXTZip: Compress and Decompress Strings with Gzip Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-basics.md Provides functionality to compress and decompress strings using the gzip algorithm. It accepts a boolean parameter 'unzip' to toggle between compression and decompression. The output is typically a Buffer. ```javascript const zlib = require('zlib'); function TXTZip(str, unzip = false) { if (unzip) { return zlib.gunzipSync(Buffer.from(str, 'base64')); } else { return zlib.gzipSync(str); } } ``` -------------------------------- ### Binary Dependent Variable Definition in Econometrics Source: https://github.com/inist-cnrs/ezs/blob/master/packages/teeft/examples/data/en-articles/STI2012-DCM-model.txt Defines a binary dependent variable for a statistical model, where 'one' indicates acceptance and 'zero' indicates rejection of a research proposal. This is a common setup in binary classification problems within econometrics. ```latex Y_i = 1 \text{ if a proposal is accepted (and zero otherwise), given by} ``` -------------------------------- ### Fetch URL Content and Add to Object (URLFetch) Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md URLFetch fetches content from a URL and adds it as a new field to an existing object, or returns the content directly if no target object is specified. This function is useful for enriching data with external resources. Parameters include 'url', 'target', and 'key'. ```javascript const fetch = require('node-fetch'); async function urlFetch(params) { const { url, target, key } = params; if (!url) { throw new Error('URL is required'); } try { const response = await fetch(url); if (!response.ok) { throw new Error(`HTTP error! status: ${response.status}`); } const content = await response.text(); if (target && key) { target[key] = content; return target; } else { return content; } } catch (error) { throw error; } } ``` -------------------------------- ### List Files Matching Pattern using JavaScript Source: https://github.com/inist-cnrs/ezs/blob/master/packages/teeft/README.md Generates a list of file paths that match a given pattern within specified directories. Supports wildcard patterns for flexible file searching. Returns an array of strings, where each string is a file path. ```javascript /* * Lists files matching a pattern in given directories. * Parameters: * pattern: File pattern (e.g., "*.txt") (default "*"). * Returns: Array of file paths (string). */ const filePaths = TeeftListFiles(['/path/to/dir1', '/path/to/dir2'], '*.js'); ``` -------------------------------- ### Distribute Data by ID and Value Source: https://github.com/inist-cnrs/ezs/blob/master/packages/analytics/README.md The 'distribute' plugin takes an array of objects, each with an 'id' and 'value', and generates a series of numbers based on the provided 'value'. It interpolates missing values and allows customization of step, start, size, and default values. This is useful for creating sequential data distributions. ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2003, "value": 3 }, { "id": 2005, "value": 4 }, { "id": 2007, "value": 5 }, { "id": 2009, "value": 6 }, { "id": 2011, "value": 7 }, { "id": 2013, "value": 8 }, ] ``` ```json [ { "id": 2000, "value": 1 }, { "id": 2001, "value": 2 }, { "id": 2002, "value": 0 }, { "id": 2003, "value": 3 }, { "id": 2004, "value": 0 }, { "id": 2005, "value": 4 }, { "id": 2006, "value": 0 }, { "id": 2007, "value": 5 }, { "id": 2008, "value": 0 }, { "id": 2009, "value": 6 }, { "id": 2010, "value": 0 }, { "id": 2011, "value": 7 }, { "id": 2012, "value": 0 }, { "id": 2013, "value": 8 } ] ``` -------------------------------- ### URLRequest API Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md Fetches data from a given URL and optionally processes the result as JSON. ```APIDOC ## URLRequest ### Description Takes an `Object` as parameters for a URL, sends a request, and returns the result. ### Method POST (implied by usage, typically used within a larger request context) ### Endpoint `/api/url/request` (hypothetical endpoint based on description) ### Parameters #### Query Parameters - **url** (String) - Required - The URL to fetch. - **json** (Boolean) - Optional - Parse result as JSON. Defaults to `true`. - **target** (String) - Optional - Choose the key to set in the result. - **timeout** (Number) - Optional - Timeout in milliseconds for the request. Defaults to `1000`. - **noerror** (Boolean) - Optional - Ignore all errors; the target field will remain undefined. Defaults to `false`. - **retries** (Number) - Optional - The maximum number of times to retry the connection. Defaults to `5`. - **insert** (String) - Optional - A header response value to insert into the result. #### Request Body - **Input Data** - The input data to be sent with the request, often an array of objects. ### Request Example ```json { "url": "https://api.search.net", "json": true, "target": "result" } ``` ### Response #### Success Response (200) - **Object** - The result of the URL fetch, potentially parsed as JSON. #### Response Example ```json [ { "result": "a" } ] ``` ``` -------------------------------- ### Metrics Command (Placeholder) Source: https://github.com/inist-cnrs/ezs/blob/master/docs/plugin-core.md The 'metrics' command is used for collecting metrics within the ezs server environment when EZS_METRICS is enabled. It passes objects through without modification but advises against using 'input' or 'output' as bucket names. ```string metrics ``` -------------------------------- ### Data Grouping and Ungrouping using INI directives Source: https://context7.com/inist-cnrs/ezs/llms.txt This demonstrates the 'group' and 'ungroup' directives for managing data streams in batches. The 'group' directive collects individual items into arrays of a specified size, while 'ungroup' reverses this process, expanding arrays back into individual items. The provided input and output examples illustrate the behavior of these directives. ```ini # Group items into batches of 3 [group] size = 3 # Process batch... # Ungroup back to individual items [ungroup] ``` -------------------------------- ### Data Expansion and Enrichment using 'expand' directive in INI Source: https://context7.com/inist-cnrs/ezs/llms.txt This example shows how to enrich data by expanding field values using external lookups. The 'expand' directive, utilizing the 'analytics' plugin, replaces a field's value (e.g., department code) with corresponding data from a specified lookup file, providing a way to add context or detailed information to the dataset. ```ini [use] plugin = analytics # Expand department codes to full names [expand] path = dept size = 1 file = ./lookup-department.ini ``` -------------------------------- ### URLParse API Source: https://github.com/inist-cnrs/ezs/blob/master/packages/basics/README.md Parses a given URL string and returns an object containing its components. ```APIDOC ## URLParse ### Description Takes a URL `String`, parses it, and returns an `Object` containing its components. ### Method POST (implied by usage, typically used within a larger request context) ### Endpoint `/api/url/parse` (hypothetical endpoint based on description) ### Parameters #### Request Body - **url** (String) - Required - The URL string to parse. ### Request Example ```json { "url": "https://example.com:8080/path/to/resource?query=test#hash" } ``` ### Response #### Success Response (200) - **Object** - An object with the following fields: - `href` (String) - The full URL. - `origin` (String) - The URL's origin (protocol + host). - `protocol` (String) - The URL's protocol. - `username` (String) - The URL's username. - `password` (String) - The URL's password. - `host` (String) - The URL's host (hostname + port). - `hostname` (String) - The URL's hostname. - `port` (String) - The URL's port. - `pathname` (String) - The URL's path. - `search` (String) - The URL's query string (including '?'). - `hash` (String) - The URL's fragment identifier (including '#'). #### Response Example ```json { "href": "https://example.com:8080/path/to/resource?query=test#hash", "origin": "https://example.com:8080", "protocol": "https:", "username": "", "password": "", "host": "example.com:8080", "hostname": "example.com", "port": "8080", "pathname": "/path/to/resource", "search": "?query=test", "hash": "#hash" } ``` ```