### Starting Ulixee Cloud Node Programmatically

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/overview/introduction.md

This JavaScript example demonstrates how to programmatically start a Ulixee Cloud node using the `CloudNode` class from the `@ulixee/cloud` package. It initializes a new `CloudNode` instance and then calls its `listen()` method to start the server, which returns a Promise.

```javascript
import { CloudNode } from '@ulixee/cloud';

(async () => {
  const cloudNode = new CloudNode();
  await cloudNode.listen();
})();
```

--------------------------------

### Starting a Ulixee Cloud Instance Programmatically

Source: https://github.com/ulixee/platform/blob/main/cloud/README.md

This JavaScript example demonstrates how to programmatically initialize and start a Ulixee Cloud instance. It imports the `Cloud` class, creates a new instance, and then calls the `listen()` method to begin accepting connections.

```javascript
import { Cloud } from '@ulixee/cloud';

(async () => {
  const cloud = new Cloud();
  await cloud.listen();
})();
```

--------------------------------

### Installing Docker on Ubuntu

Source: https://github.com/ulixee/platform/blob/main/cloud/tools/docker/README.md

This script installs Docker on an Ubuntu system, configures it to start automatically on boot, and grants the current user non-root access to Docker commands, requiring a system reboot to apply changes.

```bash
sudo apt update
sudo apt install docker.io
# start a daemon to auto-run service
sudo systemctl start docker.service
sudo systemctl enable docker.service
# allow non-root access
sudo usermod -aG docker $USER
reboot
```

--------------------------------

### Starting an Ulixee Datastore - npx

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/introduction.md

These commands demonstrate how to start a Datastore from the command line using `npx`. The first command starts the Datastore at the specified path, while the second command includes the `--watch` flag to automatically restart the Datastore on file changes during development.

```bash
npx @ulixee/datastore start <path to datastore>

# or watching changes
npx @ulixee/datastore start --watch <path to datastore>
```

--------------------------------

### Starting Ulixee Datastore for Development via Ulixee CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Starts a pre-packaged Ulixee Datastore for local development using `ulixee datastore start`. This command simplifies querying during development by using a single, temporary version hash. It can optionally watch files for changes to automatically push new versions.

```bash
ulixee datastore start [path to pre-packaged datastore]
```

--------------------------------

### Starting Cloud Node and Creating Argon Localchain - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/configuration.md

This command starts an Ulixee Cloud Node and instructs it to automatically create an Argon localchain if one does not already exist. It also specifies the Argon mainchain URL, simplifying the setup process for payment services.

```bash
npx @ulixee/cloud start \
    --argon-localchain-create-if-missing \
    --argon-mainchain-url wss://rpc.testnet.argonprotocol.org
```

--------------------------------

### Starting Argon Localchain with Ulixee Cloud - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/setup-datastore-payments.md

This command starts the Ulixee CloudNode and automatically creates a 'primary' Argon Localchain if one doesn't already exist. The Localchain is essential for processing payments within the Ulixee ecosystem.

```bash
npx @ulixee/cloud start --argon-localchain-create-if-missing
```

--------------------------------

### Displaying Cloud Node Start Help - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/configuration.md

This command displays the available command-line options and help information for starting an Ulixee Cloud Node using the `@ulixee/cloud` CLI. It's useful for understanding various configuration flags.

```bash
npx @ulixee/cloud start --help
```

--------------------------------

### Creating a Basic Ulixee Datastore with an Extractor (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/datastore.md

This snippet demonstrates the basic instantiation of a `Datastore` object in Ulixee. It shows how to define a simple extractor named `nameOfExtractor` that emits a message using input data. This example highlights the minimal setup required to get a datastore running with an extractor.

```JavaScript
import Datastore from '@ulixee/datastore';

export default new Datastore({
  extractors: {
    nameOfExtractor: new Extractor(extractorContext => {
      extractorContext.Output.emit({ message: `Hello ${extractorContext.input.firstName}` });
    }),
  },
});
```

--------------------------------

### Installing Ulixee Cloud Node Dependencies - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/setup-datastore-payments.md

This command installs the core Ulixee Cloud package and its dependencies, including @argonprotocol/localchain for payments and @ulixee/datastore for Datastore functionality, which are prerequisites for setting up Datastore payments.

```bash
npm install @ulixee/cloud
```

--------------------------------

### Starting Ulixee Datastore for Development via npx CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Starts a pre-packaged Ulixee Datastore for local development using `npx @ulixee/datastore start`. This command simplifies querying during development by using a single, temporary version hash. It can optionally watch files for changes to automatically push new versions.

```bash
npx @ulixee/datastore start [path to pre-packaged datastore]
```

--------------------------------

### Installing Ulixee Cloud with npm

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/overview/introduction.md

This command installs the `@ulixee/cloud` package and its dependencies (Hero, Datastore, Desktop Core) into your project using npm. The `--save` flag adds it to your project's `dependencies` in `package.json`.

```bash
npm i --save @ulixee/cloud
```

--------------------------------

### Installing Ulixee Cloud with npm

Source: https://github.com/ulixee/platform/blob/main/cloud/README.md

This command installs the `@ulixee/cloud` package and its core dependencies, including Hero, Datastore, and Desktop Core, into your project using the npm package manager.

```bash
npm i --save @ulixee/cloud
```

--------------------------------

### Installing Ulixee Datastore NPM Package

Source: https://github.com/ulixee/platform/blob/main/datastore/README.md

This command installs the `@ulixee/datastore` package as a dependency in your project, enabling you to use Datastore functionalities. It is a prerequisite for developing Ulixee Datastores and should be run in your project's root directory.

```bash
npm i --save @ulixee/datastore
```

--------------------------------

### Starting a Ulixee Datastore Locally using npx

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/passthrough-extractor.md

This Bash command demonstrates how to start a Ulixee Datastore from a local JavaScript file using `npx`. It initializes the datastore, making its extractors available for querying.

```Bash
npx @ulixee/datastore start ./example.js
```

--------------------------------

### Starting Ulixee Cloud with Hosted Services and Localchain (Lead Node)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/payments.md

This command initializes the Ulixee Cloud as a lead node, enabling the EmbeddedPaymentService. It configures a specific port for hosted services, defines the localchain path, sets the mainchain URL for Argon Protocol, and specifies the block rewards address. This setup is crucial for managing micropayment channels for cloned Datastores.

```bash
$ npx @ulixee/cloud start --hosted-services-port 18181 \
    --argon-localchain-path /path/to/localchain \
    --argon-mainchain-url wss://rpc.testnet.argonprotocol.org \
    --argon-block-rewards-address 5DRTmdnaztvtdZ56QbEmHM8rqUR2KiKh7KY1AeMfyvkPSb5S
```

--------------------------------

### Starting Ulixee Cloud as a Child Node

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/payments.md

This command starts the Ulixee Cloud as a child node, connecting it to a pre-configured lead node (Hosted Services node). The `--setup-host` flag directs the child node to the specified host and port of the lead node, allowing it to inherit payment service configurations and participate in the cluster.

```bash
$ npx @ulixee/cloud start --setup-host <host ip>:18181
```

--------------------------------

### Installing Ulixee Datastore - npm

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/introduction.md

This command installs the @ulixee/datastore package into your project using npm. It's a prerequisite for developing Datastores. Ensure your Ulixee development environment, including @ulixee/cloud, is already set up and running.

```bash
npm i --save @ulixee/datastore
```

--------------------------------

### Verifying Node.js Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/help/prerequisites.md

This command is used to check if Node.js has been successfully installed on your system. It outputs the installed version number of Node.js, confirming its presence and readiness for use.

```Shell
node --version
```

--------------------------------

### Verifying Node.js Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/cli/docs/help/prerequisites.md

This command checks the installed version of Node.js, confirming a successful installation. A version number should be displayed as output.

```Shell
node --version
```

--------------------------------

### Verifying Node.js Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/client/docs/help/prerequisites.md

This command is used to check the installed version of Node.js, confirming a successful installation. The expected output is a version number.

```Shell
node --version
```

--------------------------------

### Defining npm Script for Ulixee Cloud CLI

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/overview/introduction.md

This JSON snippet shows how to add a custom script to your `package.json` file. The `ulixee-start` script executes the `@ulixee/cloud start` command, allowing you to launch a Ulixee Cloud instance conveniently via `npm run ulixee-start`.

```json
{
  "scripts": {
    "ulixee-start": "@ulixee/cloud start"
  }
}
```

--------------------------------

### Starting Ulixee Server using npx

Source: https://github.com/ulixee/platform/blob/main/sql/docs/clients/postgres.md

This shell command starts the Ulixee Server playground, which is a prerequisite for running Ulixee SQL queries. It uses 'npx' to execute the '@ulixee/server-playground' package, making the server available for client connections.

```Shell
% npx @ulixee/server-playground start
```

--------------------------------

### Starting Ulixee Databroker Server with Options - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/databrokers.md

This command starts the Ulixee Databroker server, specifying various configuration options such as the public API port, hostname, admin interface port, storage directory for databases, and the localchain data path. These options allow customization of the Databroker's operation and data persistence.

```bash
$ @ulixee/databroker start --port 8080 --hostname localhost --admin-port 8081 \
  --storage-dir /path/to/storage --localchain-path /path/to/localchain
```

--------------------------------

### Configuring npm Script for Ulixee Cloud CLI

Source: https://github.com/ulixee/platform/blob/main/cloud/README.md

This JSON snippet illustrates how to add a custom script to your `package.json` file, enabling you to start the Ulixee Cloud instance conveniently from the command line using `npm run ulixee-start`.

```json
{
  "scripts": {
    "ulixee-start": "@ulixee/cloud start"
  }
}
```

--------------------------------

### Verifying npm Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/help/prerequisites.md

This command verifies the installation of npm (Node Package Manager), which is bundled with Node.js. It displays the installed npm version, indicating that the package manager is available for managing project dependencies.

```Shell
npm --version
```

--------------------------------

### Installing Remote Ulixee Datastore Locally via npx CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Installs a remote Ulixee Datastore into the local project using `npx @ulixee/datastore install <datastoreId> <version>`. This command adds local TypeScript typings for the specified Datastore, making its types available for use in the project.

```bash
npx @ulixee/datastore install <datastoreId> <version>
```

--------------------------------

### Publishing Built Packages with Lerna (Shell)

Source: https://github.com/ulixee/platform/blob/main/RELEASE-CHECKLIST.md

This sequence of commands first changes the directory to `build-dist`, then uses Lerna to publish packages from their `package.json` files. This step is for publishing the previously built distribution files.

```Shell
cd build-dist && lerna publish from-package
```

--------------------------------

### Starting Cloud Node with Existing Argon Localchain - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/configuration.md

This command starts an Ulixee Cloud Node, configuring it to connect to an existing Argon localchain at a specified path. It also sets the mainchain URL and prompts for the localchain password interactively, ensuring secure access to payment processing.

```bash
npx @ulixee/cloud start \
    --argon-localchain-path /path/to/localchain \
    --argon-mainchain-url wss://rpc.testnet.argonprotocol.org\
    --argon-localchain-password-interactive
```

--------------------------------

### Networking Ulixee CloudNodes for Service Discovery in JavaScript

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/modules/cloud-node.md

This example illustrates how to network two Ulixee CloudNodes, configuring one as a services host and the other as a client. The `servicesNode` activates hosted services on port 18181, and the `childNode` connects to it for service discovery using the `servicesSetupHost` option. This setup enables delegation of storage and discovery tasks within a cloud cluster.

```JavaScript
import { CloudNode } from '@ulixee/cloud';

(async () => {
  const servicesNode = new CloudNode({
    port: 1818,
    hostedServicesServerOptions: { port: 18181 }, // default services port
  });
  await servicesNode.listen();

  const childNode = new CloudNode({
    // resolves to localhost:18181
    servicesSetupHost: await servicesNode.hostedServicesServer.host,
  });
  await childNode.listen();
})().catch(error => {
  console.log('ERROR starting Ulixee CloudNode', error);
  process.exit(1);
});
```

--------------------------------

### Installing Remote Ulixee Datastore Locally via Ulixee CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Installs a remote Ulixee Datastore into the local project using `ulixee datastore install <datastoreId> <version>`. This command adds local TypeScript typings for the specified Datastore, making its types available for use in the project.

```bash
ulixee datastore install <datastoreId> <version>
```

--------------------------------

### Verifying npm Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/client/docs/help/prerequisites.md

This command is used to check the installed version of npm (Node Package Manager), which is typically bundled with Node.js. The expected output is a version number.

```Shell
npm --version
```

--------------------------------

### Seeding Ulixee Table Data with onCreated (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/table.md

This example illustrates how to use the `onCreated` callback within a Table definition inside a Datastore. The `onCreated` function is executed once when the Table is installed, allowing for initial data population using `insertInternal`.

```JavaScript
import Datastore from '@ulixee/datastore';

const whitelist = new Set([`id1xv7empyzlwuvlshs2vlf9eruf72jeesr8yxrrd3esusj75qsr6jqj6dv3p`]);

export default new Datastore({
  tables: {
    events: new Table({
      schema: {
        name: string(),
        date: date(),
      },
    }),
    async onCreated() {
      await this.insertInternal({
        name: 'Thanksgiving',
        birthdate: new Date('2023-11-01'),
      });
    },
  },
});
```

--------------------------------

### Building Distribution Files with Yarn (Shell)

Source: https://github.com/ulixee/platform/blob/main/RELEASE-CHECKLIST.md

This command executes the `build:dist` script defined in the project's `package.json` using Yarn. Its purpose is to compile and prepare the distribution-ready files for the project.

```Shell
yarn build:dist
```

--------------------------------

### Pushing Git Tags to Origin (Shell)

Source: https://github.com/ulixee/platform/blob/main/RELEASE-CHECKLIST.md

This command pushes all local Git tags to the `main` branch on the `origin` remote. Pushing tags is crucial as it triggers the building of release assets in the CI/CD pipeline.

```Shell
git push origin main --tags
```

--------------------------------

### Using Data Utilities in ExtractorSchema Input Examples

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/extractor-schemas.md

This snippet demonstrates how to use built-in data utility functions like `dateAdd` within `inputExamples` of an `ExtractorSchema`. It shows how to define dynamic example input values for testing and documentation purposes, ensuring examples are always relevant.

```js
import { Extractor, HeroExtractorPlugin } from '@ulixee/datastore-plugins-hero';
import { string, dateAdd } from '@ulixee/schema';

export default new Extractor({
  async run(ctx) {
    // prints 'YYYY-MM-DD' of tomorrow
    console.log(ctx.input.when);
  },
  schema: {
    input: {
      when: string({ format: 'date' }),
    },
    inputExamples: [{ when: (1, 'days') }],
  },
});
```

--------------------------------

### Verifying npm Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/cli/docs/help/prerequisites.md

This command checks the installed version of npm (Node Package Manager), which is bundled with Node.js. A version number should be displayed as output.

```Shell
npm --version
```

--------------------------------

### Verifying Node.js Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/help/prerequisites.md

This command is used to verify that Node.js has been successfully installed on the system by displaying its version number. It's a crucial step to ensure the environment is ready for Ulixee Cloud development.

```Shell
node --version
```

--------------------------------

### Deploying Ulixee Datastore to CloudNode - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/setup-datastore-payments.md

This command deploys a Ulixee Datastore (assuming datastore.ts) to a specified CloudNode host. Successful deployment requires port 1818 to be publicly available on the server hosting the CloudNode.

```bash
npx @ulixee/cloud deploy ./datastore.ts \
  --cloud-host <your-host>:1818
```

--------------------------------

### Running Yarn for Dependency Management - Yarn

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/contribute/how-to.md

The `yarn` command is used to install and manage project dependencies. It's specifically mentioned for cleaning up the `yarn.lock` file by removing temporary project dependencies before committing, and for re-initializing dependencies after project structure changes.

```Bash
yarn
```

--------------------------------

### Installing Ulixee Databroker Server - Bash

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/databrokers.md

This command installs the Ulixee Databroker CLI package using npm. It is a prerequisite for running a Databroker server, which handles Argon Localchain payments and data access at a client level.

```bash
npm install @ulixee/databroker
```

--------------------------------

### Bumping Prerelease Version with ulx-repo-version-bump (Shell)

Source: https://github.com/ulixee/platform/blob/main/RELEASE-CHECKLIST.md

This command uses the `ulx-repo-version-bump` utility to increment the prerelease version of packages. It should be replaced with a standard bump if not in a prerelease phase.

```Shell
npx ulx-repo-version-bump prerelease
```

--------------------------------

### Connecting Hero Client to Ulixee Cloud

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/overview/introduction.md

This JavaScript snippet illustrates how to connect a Hero client to a running Ulixee Cloud instance. It imports the `Hero` class and creates an instance, explicitly specifying the `connectionToCore` option with the Cloud's WebSocket address. The example then navigates to a URL and closes the Hero instance.

```javascript
import Hero from '@ulixee/hero';

(async () => {
  const hero = new Hero({ connectionToCore: 'ws://localhost:1818' });
  await hero.goto('https://example.org');
  await hero.close();
})();
```

--------------------------------

### Initializing a Basic Ulixee CloudNode in JavaScript

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/modules/cloud-node.md

This snippet demonstrates how to initialize and start a basic Ulixee CloudNode instance. It configures the node to listen on port 1818 and logs the active port once the node is successfully listening. It includes error handling for the startup process.

```JavaScript
import { CloudNode } from '@ulixee/cloud';

(async () => {
  const cloudNode = new CloudNode({
    port: 1818,
  });
  await cloudNode.listen();

  console.log(`CloudNode started on port ${await cloudNode.port}`);
})().catch(error => {
  console.log('ERROR starting Ulixee CloudNode', error);
  process.exit(1);
});
```

--------------------------------

### Connecting Hero to a Ulixee Cloud Instance

Source: https://github.com/ulixee/platform/blob/main/cloud/README.md

This JavaScript example shows how to connect a Hero client to a running Ulixee Cloud instance by specifying the `connectionToCore` option in the Hero constructor. It then navigates to a URL and ensures the Hero instance is properly closed.

```javascript
import Hero from '@ulixee/hero';

(async () => {
  const hero = new Hero({ connectionToCore: 'ws://localhost:1818' });
  await hero.goto('https://example.org');
  await hero.close();
})();
```

--------------------------------

### Updating Dependencies with ulx-repo-version-check (Shell)

Source: https://github.com/ulixee/platform/blob/main/RELEASE-CHECKLIST.md

Executed from the `@ulixee/ulixee` package, this command uses `ulx-repo-version-check` with the `fix` option to update and synchronize project dependencies, ensuring they align with the latest versions.

```Shell
npx ulx-repo-version-check fix
```

--------------------------------

### Example Ulixee Zone Record Configuration (JSON)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/register-a-domain.md

This JSON snippet illustrates the structure of a Ulixee Zone Record. It includes a 'paymentAccount' for transactions, a 'notaryId', and 'versions' which map specific software versions to their 'host' (e.g., a WebSocket URL) and associated 'datastoreId'. Users should replace the placeholder values with their actual Datastore Payments account and update version details.

```json
{
  "paymentAccount": "5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889",
  "notaryId": 1,
  "versions": {
    "1.0.0": {
      "host": "wss://206.189.181.34:1818",
      "datastoreId": "vegetarian-recipes"
    }
  }
}
```

--------------------------------

### Verifying npm Installation - Shell

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/help/prerequisites.md

This command checks the installed version of npm (Node Package Manager), which is bundled with Node.js. A successful output confirms that npm is available for managing project dependencies for Ulixee Cloud.

```Shell
npm --version
```

--------------------------------

### Upgrading Ulixee Hero to Latest Version (Shell)

Source: https://github.com/ulixee/platform/blob/main/cloud/docs/help/how-to-upgrade.md

This command upgrades the `@ulixee/cloud` package, which includes the Hero component, to its latest available version. It ensures that your local Ulixee installation is up-to-date with the newest features and bug fixes. This command should be run in your project's root directory or globally if `@ulixee/cloud` is installed globally.

```Shell
npm install @ulixee/cloud@latest
```

--------------------------------

### Seeding Ulixee Datastore Tables on Creation - JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/datastore.md

This JavaScript snippet illustrates the use of the `onCreated` callback to bootstrap an Ulixee Datastore. It defines a table schema and inserts initial data into an internal table when the Datastore is first installed.

```javascript
import Datastore from '@ulixee/datastore';

const whitelist = new Set([`id1xv7empyzlwuvlshs2vlf9eruf72jeesr8yxrrd3esusj75qsr6jqj6dv3p`]);

export default new Datastore({
  tables: {
    events: new Table({
      schema: {
        name: string(),
        date: date()
      }
    })
  },
  async onCreated() {
    await this.tables.events.insertInternal({
      name: 'Thanksgiving',
      birthdate: new Date('2023-11-01')
    });
  }
});
```

--------------------------------

### Pulling Ulixee Docker Image from Dockerhub

Source: https://github.com/ulixee/platform/blob/main/cloud/tools/docker/README.md

This command sequence pulls the latest Ulixee Cloud Docker image from Dockerhub and then tags it with a shorter, more convenient name 'ulixee-cloud' for local use.

```bash
docker pull ulixee/ulixee-cloud:latest
docker tag ulixee/ulixee-cloud ulixee-cloud
```

--------------------------------

### Implementing Ulixee Datastore Identity Authentication - JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/datastore.md

This JavaScript example demonstrates how to implement the `authenticateIdentity` callback in an Ulixee Datastore. It uses a whitelist to restrict access to specific pre-approved identities, enhancing Datastore security.

```javascript
import Datastore from '@ulixee/datastore';

const whitelist = new Set([`id1xv7empyzlwuvlshs2vlf9eruf72jeesr8yxrrd3esusj75qsr6jqj6dv3p`]);

export default new Datastore({
  authenticateIdentity(identity, nonce) {
    return whitelist.has(identity);
  }
});
```

--------------------------------

### Initializing a Crawler with HeroExtractorPlugin (Single-Stage) - JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/hero-plugin.md

This snippet demonstrates how to create a basic Ulixee Datastore Crawler using the HeroExtractorPlugin. It performs a live browser interaction to navigate to a URL, extract the page title and body content, and then returns the Hero instance. This setup is suitable for single-stage extraction where data is processed immediately after collection.

```JavaScript
import { HeroExtractorPlugin, Crawler } from '@ulixee/datastore-plugins-hero';
export default new Crawler(async context => {
  const { input, Output, Hero } = context;

  const hero = new Hero();
  await hero.goto(input.url);
  const title = await hero.document.title;

  const output = new Output();
  output.title = title;
  output.body = await hero.document.body.textContent;
  return hero;
}, HeroExtractorPlugin);
```

--------------------------------

### Querying Datastore via Testnet Databroker - TypeScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/querying-datastores.md

This example illustrates querying a Ulixee Datastore using the `@ulixee/client` library through the Ulixee Foundation's Testnet Databroker. It connects to 'Meals.Health' v0.0.1, queries for 'paleo' recipes, and logs the output. The `DefaultPaymentService.fromBroker()` is configured with the Databroker's WebSocket URL and requires a path to an Identity PEM file for authentication.

```TypeScript
import { Client, DefaultPaymentService } from '@ulixee/client';

(async () => {
  const client = new Client(`ulx://Meals.Health/v0.0.1`, {
    paymentService: await DefaultPaymentService.fromBroker(
      'wss://databroker.testnet.ulixee.org',
      {
        pemPath: 'path to your Identity pem file',
      },
    ),
  });
  const results = await client.query(
    `SELECT * from recipes where diet = 'paleo`,
  );

  console.log(results);

  await client.disconnect();
})().catch(console.error);
```

--------------------------------

### Building Datastores to a Specific Output Directory using CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

This command allows users to build Ulixee Datastores and specify a custom output directory for the generated .dbx folders. This is useful for pre-deploying .dbx files, for instance, in a Docker image setup.

```Shell
npx @ulixee/datastore build --out-dir=<path>
```

--------------------------------

### Configuring Puppeteer Launch Options in Ulixee Extractor

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/puppeteer-plugin.md

This example illustrates how to configure Puppeteer launch options when running an Ulixee Extractor. It shows a basic Extractor using `ctx.browser.newPage()` and extracting a page title, then demonstrates calling `runInternal` with `waitForInitialPage: false`.

```JavaScript
import Datastore from '@ulixee/datastore';
import { Extractor, PuppeteerExtractorPlugin } from '@ulixee/datastore-plugins-puppeteer';

const datastore = new Datastore({
  extractors: {
    pupp: new Extractor(async ctx => {
      const page = await ctx.browser.newPage();
      output.title = await page.evaluate(() => {
        return document.querySelector('title').textContent;
      });
    }, PuppeteerExtractorPlugin),
  },
});
await datastore.extractors.pupp.runInternal({ waitForInitialPage: false });
```

--------------------------------

### Pulling Ulixee Docker Image from GitHub Container Registry

Source: https://github.com/ulixee/platform/blob/main/cloud/tools/docker/README.md

This command sequence pulls the latest Ulixee Cloud Docker image from the GitHub Container Registry and then tags it with a shorter, more convenient name 'ulixee-cloud' for local use.

```bash
docker pull ghcr.io/ulixee/ulixee-cloud:latest
docker tag ghcr.io/ulixee/ulixee-cloud:latest ulixee-cloud
```

--------------------------------

### Setting Base Price for Ulixee Datastore Extractor - TypeScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/setup-datastore-payments.md

This TypeScript code defines a Ulixee Datastore with an extractor named 'basic' and sets its basePrice to 1,000 Microgons (equivalent to 1 milligon or 1/1000 of a penny). This configures the Datastore to charge for data extraction.

```typescript
import Datastore, { Extractor } from '@ulixee/datastore';

const datastore = new Datastore({
  name: 'HelloWorld',
  extractors: {
    // Add a price of 1 milligon (a penny)
    basic: new Extractor({
      basePrice: 1_000,
      async run({ Output }) {
        for (const line of ['hello', 'world']) {
          Output.emit({
            line,
          });
        }
      },
    }),
  },
});

export default datastore;
```

--------------------------------

### Initializing Ulixee Client with Payment Service (TypeScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/interface/remote-client.md

This example shows how to initialize an Ulixee Client using a connection URI and integrating a DefaultPaymentService. It configures the client with a payment service obtained from a databroker and specifies an Argon Mainchain URL for notary lookups, enabling micropayment channel transactions.

```typescript
import { DefaultPaymentService } from '@ulixee/databroker';
import Client from '@ulixee/client';

const paymentService = await DefaultPaymentService.fromBroker('wss://broker.testnet.ulixee.org', {
  pemPath: 'path to your Identity pem file',
});
const client = new Client('ulx://UsCPI.Stats/v1.0.0', {
  paymentService,
  argonMainchainUrl: 'wss://rpc.testnet.argonprotocol.org',
});
```

--------------------------------

### Implementing a Custom Plugin Class (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/plugins.md

This snippet provides an example of a custom plugin class implementation. It defines `name` and `version` properties and an asynchronous `run` method. Inside `run`, it demonstrates how to extend the `context` (e.g., by adding a `Hero` constructor) and how to properly use the `next()` callback to wait for the extractor's completion. The `finally` block ensures resources like `hero` and `heroReplay` are properly closed, highlighting the importance of cleanup in plugin development.

```javascript
class Plugin {
  name = pkg.name;
  version = pkg.version;

  async run(extractorInternal, context, next) {
    try {
      context.Hero = createBoundHeroConstructor();
      //  wait for next to complete
      const output = await next();
    } finally {
      // cleanup
      await this.hero?.close();
      await this.heroReplay?.close();
    }
  }
}
```

--------------------------------

### Migrating Ulixee Table Data with onVersionMigrated (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/table.md

This snippet demonstrates the `onVersionMigrated` callback, which is invoked when a new version of a Table is installed and a previous version exists. It shows how to fetch data from the `previousTable` and insert it into the new table, useful for schema evolution and data migration.

```JavaScript
import Datastore from '@ulixee/datastore';

export default new Datastore({
  tables: {
    events: new Table({
      schema: {
        name: string(),
        reason: string({ enum: ['fun', 'work'] }),
        date: date(),
      },
      async onVersionMigrated(previousTable) {
        const previousEvents = await previousTable.fetchInternal();
        for (const previousEvent of previousEvents) {
          await this.insertInternal(
            ...previousEvents.map(x => ({
              ...x,
              reason: 'fun',
            })),
          );
        }
      },
    }),
  },
});
```

--------------------------------

### Deploying and Running a Ulixee Datastore on Cloud

Source: https://github.com/ulixee/platform/blob/main/datastore/README.md

These bash commands demonstrate how to deploy a Datastore script to an Ulixee Cloud and then run it remotely. The first command deploys `simple.js` to the cloud, and the second executes it on the cloud, passing the `firstName` input parameter.

```bash
npx @ulixee/datastore deploy ./simple.js
npx @ulixee/datastore run simple.js --input.firstName=Me
```

--------------------------------

### Configuring a Ulixee Datastore with ID, Version, and Schema-Defined Extractor (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/datastore.md

This example demonstrates a more detailed `Datastore` configuration, including explicit `id` and `version` properties. It defines an `Extractor` with a `run` method that processes input and an associated `schema` for both input (`url`) and output (`urlLength`), ensuring data validation and structure.

```JavaScript
import Datastore, { Extractor } from '@ulixee/datastore';

export default new Datastore({
  id: 'test',
  version: '0.0.1',
  extractors: {
    instance: new Extractor({
      run({ input, Output }) {
        const output = new Output();
        output.urlLength = input.url.length;
      },
      schema: {
        input: {
          url: string({ format: 'url' }),
        },
        output: {
          urlLength: number(),
        },
      },
    }),
  },
});
```

--------------------------------

### Interacting with Local Datastores using Ulixee Client (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/overview/introduction.md

This snippet demonstrates how to use Ulixee Client with a locally defined `Table` for development purposes. It sets up a schema, seeds initial data, and then queries the local table using SQL with a parameterized input.

```javascript
import Client from '@ulixee/client';
import { Table, string } from '@ulixee/datastore';

const testingTable = new Table({
  schema: {
    firstName: string(),
    lastName: string(),
  },
  seedlings: [
    { firstName: 'Dennis', lastName: 'Rodman' },
    { firstName: 'Michael', lastName: 'Jordan' },
  ],
});

const client = new Client(testingTable);
client.query('SELECT * FROM self WHERE lastName=$1', ['Jordan']).then(records => {
  console.log(records);
});
```

--------------------------------

### Querying Remote Datastores with SQL using Ulixee Client (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/overview/introduction.md

This snippet demonstrates how to connect to a remote Ulixee Datastore using a `ulx://` URL and execute a SQL query to retrieve records. It shows the basic pattern of client initialization and calling the `query` method.

```javascript
import Client from '@ulixee/client';

const client = new Client('ulx://USERNAME:PASSWORD@DOMAIN:PORT/DATABASE');
client.query(`SELECT * FROM developers WHERE status='founder'`).then(records => {
  console.log(records);
});
```

--------------------------------

### Upgrading Ulixee Client using NPM

Source: https://github.com/ulixee/platform/blob/main/client/docs/help/how-to-upgrade.md

This command updates the @ulixee/client package to its latest available version. It's essential for accessing new features, performance improvements, and critical bug fixes. Run this command in your project's root directory or globally if the package was installed globally.

```Shell
npm install @ulixee/client@latest
```

--------------------------------

### Defining an Object Schema with Fields and Description in JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/schema.md

This example illustrates how to define an `ObjectSchema` using `object()` from `@ulixee/schema`, specifying distinct fields with their respective schema types (string and number). It includes a description for documentation purposes and demonstrates both valid and invalid data validation against the schema.

```JavaScript
import { object, string, number } from '@ulixee/schema';

const schema = object({
  description: 'Documentation that will be added to jsdocs',
  fields: {
    field1: string(),
    field2: number(),
  },
});

schema.validate({
  field1: 'test',
  field2: 1,
}); // VALID!

schema.validate({
  field1: 'test',
  field2: '1',
}); // INVALID!
```

--------------------------------

### Cloning Ulixee Platform Repository - Git

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/contribute/how-to.md

This command is used to clone the main Ulixee platform monorepo from GitHub, which is the initial step for setting up the local development environment. It retrieves all project files and history.

```Bash
git clone https://github.com/ulixee/platform.git
```

--------------------------------

### Connecting to a Ulixee Datastore via Custom Domain (TypeScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/guides/register-a-domain.md

This TypeScript snippet illustrates how a Ulixee client connects to a Datastore using its custom domain and version. Once a domain is added to a Datastore, client examples in autogenerated documentation will reflect this, allowing prospective users to easily connect using a human-readable address like 'ulx://Meals.Health/v1.0.0'.

```typescript
import Client from '@ulixee/client';

const client = new Client('ulx://Meals.Health/v1.0.0');
```

--------------------------------

### Initializing Client for Datastore in JavaScript

Source: https://github.com/ulixee/platform/blob/main/client/docs/interface/local-client.md

This snippet demonstrates how to initialize a `Client.ForDatastore` instance using a local Datastore object. It then shows how to execute a SQL query against the datastore to retrieve records, logging the results to the console. This client is used for testing an entire Datastore.

```javascript
import Client from '@ulixee/client';
import myDatastore from './myDatastore';

const client = new Client.ForDatastore(myDatastore);
client.query('SELECT firstName, lastName FROM testers WHERE lastName=$1', ['Jordan']).then(records => {
  console.log(records);
});
```

--------------------------------

### Initializing Client for Table in JavaScript

Source: https://github.com/ulixee/platform/blob/main/client/docs/interface/local-client.md

This snippet shows how to initialize a `Client.ForTable` instance using a local Table object. It then executes a SQL query against the table, using `self` as an alias, to filter records based on a bound value and logs the results. This client is designed for testing specific Tables within a Datastore.

```javascript
import Client from '@ulixee/client';
import myTable from './myTable';

const client = new Client.ForTable(testingTable);
client.query('SELECT firstName, lastName FROM self WHERE lastName=$1', ['Jordan']).then(records => {
  console.log(records);
});
```

--------------------------------

### Initializing a Simple Ulixee Datastore - JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/introduction.md

This JavaScript code defines a basic Ulixee Datastore with a single default Extractor. The Extractor takes an input `firstName` and emits a message using `ctx.Output.emit()`, demonstrating how to produce structured output from an Extractor.

```javascript
export default new Datastore({
  extractors: {
    default: new Extractor(ctx => {
      ctx.Output.emit({ message: `Hello ${ctx.input.firstName}` });
    }),
  },
});
```

--------------------------------

### Initializing a Basic Ulixee Datastore with an Extractor

Source: https://github.com/ulixee/platform/blob/main/datastore/README.md

This JavaScript code defines a simple Ulixee Datastore with a single default Extractor. The Extractor takes an input `firstName` and sets the output to a greeting string, demonstrating the basic structure of a Datastore and how to handle inputs and outputs.

```javascript
export default new Datastore({
  extractors: {
    default: new Extractor(ctx => {
      ctx.Output = `Hello ${ctx.input.firstName}`;
    }),
  },
});
```

--------------------------------

### Running a Ulixee Datastore Script Locally

Source: https://github.com/ulixee/platform/blob/main/datastore/README.md

This bash command executes the previously defined `simple.js` Datastore script directly using Node.js. It passes an input parameter `firstName` to the Datastore's default Extractor, demonstrating local execution and how to provide inputs via the command line.

```bash
node ./simple.js --input.firstName=Me
```

--------------------------------

### Initializing Ulixee Client and Querying Data (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/sql/docs/clients/ulixee.md

This snippet demonstrates how to initialize the Ulixee Client with a connection string and execute a basic SQL query. It requires the `@ulixee/client` package. The client connects to a specified Ulixee database, and the `query` method fetches data based on the provided SQL statement.

```JavaScript
import Client from '@ulixee/client';

const client = new Client('ulx://USER:PASS@HOST:PORT/DB');
const data = await client.query('SELECT * FROM table');
```

--------------------------------

### Defining an Array Schema with Element Type in JavaScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/schema.md

This example demonstrates how to create an `ArraySchema` using `array()` from `@ulixee/schema`, specifying that all elements within the array must be numbers. It includes validation examples, showing both a valid array of numbers and an invalid array containing a string.

```JavaScript
import { array, number } from '@ulixee/schema';

const schema = array({
  element: number(),
});

schema.validate([1, 2, 3]); // VALID!
schema.validate([1, 2, '3']); // INVALID!
```

--------------------------------

### Crawling Data with Ulixee Client (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/overview/no-sql.md

This snippet shows how to initiate a crawl operation on a Ulixee datastore using the `@ulixee/client`. It establishes a client connection and then invokes the 'ulixee' crawler with 'page' set to 'home' as an input. The data retrieved by the crawler is subsequently logged to the console.

```javascript
import Client from '@ulixee/client';

const client = new Client('ulx://USERNAME:PASSWORD@DOMAIN:PORT/DATABASE');
client.crawl('ulixee', { page: 'home' }).then(records => {
  console.log(records);
});
```

--------------------------------

### Building Ulixee Datastore Package via npx CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Builds a compressed Ulixee Datastore (`.dbx.tgz`) and saves it to the filesystem using `npx @ulixee/datastore build`. This is useful for deploying the same package to many environments. The Datastore is compiled into a `.dbx.build` folder, then tar-gzipped into a `.dbx` file.

```bash
npx @ulixee/datastore build [path to datastore entrypoint]
```

--------------------------------

### Deleting Rows Conditionally in SQL

Source: https://github.com/ulixee/platform/blob/main/sql/docs/sql-commands/delete.md

This example demonstrates how to delete rows from the 'films' table where the 'kind' column is not 'Musical'. It uses a WHERE clause to specify the deletion condition, ensuring only non-musical films are removed from the table.

```SQL
DELETE FROM films WHERE kind <> 'Musical';
```

--------------------------------

### Deploying a Datastore Extractor using npx

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/extractor-schemas.md

This command-line snippet shows how to deploy a Datastore Extractor using the `@ulixee/datastore` CLI tool. It packages the `index.js` file, assigning it an ID and version for deployment, making it available for consumption.

```bash
npx @ulixee/datastore deploy ./index.js; // Id=test, version=1.0.0
```

--------------------------------

### Deploying Ulixee Datastore via npx CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Deploys a Ulixee Datastore to a CloudNode using the `npx @ulixee/datastore deploy` command. Requires a path to the Datastore entrypoint, which must export an instance of `DatastoreExecutable`. The process builds and uploads the Datastore transparently without persisting local files.

```bash
npx @ulixee/datastore deploy [path to datastore entrypoint]
```

--------------------------------

### Building Ulixee Datastore Package via Ulixee CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Builds a compressed Ulixee Datastore (`.dbx.tgz`) and saves it to the filesystem using `ulixee datastore build`. This is useful for deploying the same package to many environments. The Datastore is compiled into a `.dbx.build` folder, then tar-gzipped into a `.dbx` file.

```bash
ulixee datastore build [path to datastore entrypoint]
```

--------------------------------

### Getting Ulixee Datastore Credit Balance (npx CLI)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/credits.md

This command retrieves the balance of a specific credit for a deployed Ulixee Datastore using `npx`. It requires the datastore URL and the credit ID. This command can be executed without authentication.

```bash
npx @ulixee/datastore credits get <datastore url> <credit id>
```

--------------------------------

### Deploying Ulixee Datastore via Ulixee CLI

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Deploys a Ulixee Datastore to a CloudNode using the `ulixee datastore deploy` command. Requires a path to the Datastore entrypoint, which must export an instance of `DatastoreExecutable`. The process builds and uploads the Datastore transparently without persisting local files.

```bash
ulixee datastore deploy [path to datastore entrypoint]
```

--------------------------------

### Importing Ulixee Datastore Types in JavaScript/TypeScript

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/overview/deployment.md

Demonstrates how to import and use Ulixee Datastore types in a JavaScript or TypeScript project after a Datastore has been installed locally. It shows accessing specific version and function types from the `@ulixee/datastore/types` module.

```javascript
import ITypes from '@ulixee/datastore/types';

type InputOutputDatastoreExtractorType = ITypes[version][functionName];
```

--------------------------------

### Initializing Ulixee Client with Object Configuration (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/interface/remote-client.md

This snippet demonstrates how to create a new Ulixee Client instance by providing a configuration object to its constructor. The object specifies connection details such as username, password, host, port, and database name for connecting to a remote Ulixee Datastore.

```javascript
import Client from '@ulixee/client';

const client = new Client({
  username: 'test',
  password: 'test',
  host: 'localhost',
  port: 1818,
  database: 'test',
});
```

--------------------------------

### Getting Ulixee Datastore Credit Balance (ulixee CLI)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/advanced/credits.md

This command retrieves the balance of a specific credit for a deployed Ulixee Datastore using the `ulixee` CLI. It requires the datastore URL and the credit ID. This command can be executed without authentication.

```bash
ulixee datastore credits get <datastore url> <credit id>
```

--------------------------------

### Deleting Rows and Returning All Deleted Data in SQL

Source: https://github.com/ulixee/platform/blob/main/sql/docs/sql-commands/delete.md

This example shows how to delete rows from the 'tasks' table where the 'status' is 'DONE' and simultaneously return all columns of the deleted rows. The RETURNING * clause is used to retrieve the full details of the affected records after deletion.

```SQL
DELETE FROM tasks WHERE status = 'DONE' RETURNING *;
```

--------------------------------

### Querying Remote Datastores with NoSQL using Ulixee Client (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/overview/introduction.md

This snippet illustrates connecting to a remote Ulixee Datastore and fetching records using a NoSQL-like approach with the `fetch` method. It queries a 'developers' collection based on a specified status.

```javascript
import Client from '@ulixee/client';

const client = new Client('ulx://USERNAME:PASSWORD@DOMAIN:PORT/DATABASE');
client.fetch('developers', { status: 'founders' }).then(records => {
  console.log(records);
});
```

--------------------------------

### Streaming Crawler Results with Cache Control (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/datastore/docs/basics/crawler.md

This example shows how to stream results from a Ulixee Crawler, specifying a `maxTimeInCache` of 60 seconds to control the cache age. It then uses `HeroReplay` with the `crawl` object, demonstrating how to interact with cached crawl data.

```js
const [crawl] = await crawler.stream({ input: { maxTimeInCache: 60 } });
const heroReplay = await HeroReplay(crawl);
```

--------------------------------

### Running Extractor with Ulixee Client (JavaScript)

Source: https://github.com/ulixee/platform/blob/main/client/docs/overview/no-sql.md

This snippet illustrates how to execute an extractor on a Ulixee datastore using the `@ulixee/client`. It initializes a client connection and then runs the 'lastCommit' extractor, passing 'package' as 'hero' as a parameter. The results from the extractor are logged to the console.

```javascript
import Client from '@ulixee/client';

const client = new Client('ulx://USERNAME:PASSWORD@DOMAIN:PORT/DATABASE');
client.run('lastCommit', { package: 'hero' }).then(records => {
  console.log(records);
});
```

--------------------------------

### Initializing Client for Extractor in JavaScript

Source: https://github.com/ulixee/platform/blob/main/client/docs/interface/local-client.md

This snippet illustrates how to initialize a `Client.ForExtractor` instance with a local Extractor object. It then performs a SQL query using `self` as an alias for the extractor, demonstrating how to pass bound values and log the returned records. This client is specifically for testing individual Extractors.

```javascript
import Client from '@ulixee/client';
import myExtractor from './myExtractor';

const client = new Client.ForExtractor(myExtractor);
client.query('SELECT firstName, lastName FROM self(isTesting => $1), [true]).then(records => {
  console.log(records);
});
```