IBM Aspera Validator

https://www.ibm.com/docs/en/aspera-validator/1.0.x
Admin
IBM Aspera Validator is a validation tool that ensures data integrity and compliance in file...
Tokens:12,411
Snippets:124
Trust Score:10
Update:10 months ago
Show doc for...
Tokens:
Raw
### Start Aspera Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

Starts the Aspera Validator service. This command is essential after configuration changes to apply them. Two variations are provided for different init systems (service and systemd).

```shell
# service aspera-validator start

```

```shell
# systemctl start aspera-validator

```

--------------------------------

### Start and Monitor IBM Aspera Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-installing-validator-104

These commands are used to start the OTFV validator service and to view its logs. The log viewing command differs based on the OTFV version: 'journalctl' for 1.0.6 and later, and 'tail -f' for 1.0.4 and 1.0.5. Ensure the service starts without HTTPS/certificate issues.

```bash
service aspera-validator start
```

```bash
journalctl -u aspera-validator.service -f
```

```bash
tail -f /var/log/validator.log
```

--------------------------------

### Validator Configuration: Local Server Example (YAML)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

This YAML snippet demonstrates how to configure the 'Servers' section in the validator.yaml file to include a local IBM Aspera High-Speed Transfer Server (HSTS) instance for validation. It specifies the server name, URL, and Node API credentials.

```yaml
Servers:
- Name: node0
  URL: https://localhost:9092
  User: validator_user
  Pass: sVTapkA4oBrjsjuv
```

--------------------------------

### Validator Configuration: Remote Server Example (YAML)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

This YAML snippet shows the configuration for a remote IBM Aspera High-Speed Transfer Server (HSTS) within the validator.yaml file. It includes the server's name, public URL, and the necessary Node API authentication details for the Validator to connect.

```yaml
Servers:
- Name: node1
  URL: https://node1.aspera.us:9092
  User: validator_user
  Pass: Kq4vbCBjNYdRMynN
```

--------------------------------

### Install Validator DEB Package (Bash)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

This command installs the IBM Aspera Validator using the dpkg package manager on Debian or similar Linux distributions. Replace 'version' with the correct package version.

```bash
# dpkg -i ibm-aspera-otfv-validator-version.deb
```

--------------------------------

### Enable Persistent Store with asconfigurator

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

Enables the persistent store feature for the Aspera Validator. This is crucial for maintaining state and history. If the 'persistent_store' is not set to 'enable', use this command to configure it.

```shell
# asconfigurator -x "set_central_server_data;persistent_store,enable"

```

--------------------------------

### List Existing Node API Users (Bash)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

This command lists all existing Node API users configured on the HSTS and the transfer users they are associated with. This is useful for auditing or managing existing API access.

```bash
# asnodeadmin -l
```

--------------------------------

### Start Aspera Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Command to start the Aspera Validator service using systemctl. This command is used after all configuration changes have been applied.

```bash
systemctl start aspera-validator
```

--------------------------------

### Configure Central Server Data with asuserdata

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

Sets various configuration options for the Aspera Validator's central server. This includes network settings, validation behavior, and storage options. Ensure 'schema_validation' is 'enable' for validation to function correctly.

```shell
# asuserdata -c
central server option set:
  address: "127.0.0.1"
  port: "40001"
  backlog: "200"
  schema_validation: "enable"
  raw_options: "disable"
  mgmt_backlog: "200"
  mgmt_port: "0"
  transfer_list_path: ""
  persistent_store: "enable"
  persistent_store_path: ""
  persistent_store_max_age: "86400"
  persistent_store_on_error: "ignore"
  event_buffer_capacity: "1000"
  event_buffer_overrun: "block"
  compact_on_startup: "enable"
  files_per_session: "1000000"
  file_errors: "true"
  ignore_empty_files: "true"
  ignore_skipped_files: "true"
  ignore_no_transfer_files: "true"
  db_synchronous: "off"
  db_journal: "wal"

```

--------------------------------

### Check for and Uninstall Previous OTFV Validator Versions

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-installing-validator-104

This snippet demonstrates how to check if a previous version of the OTFV validator is installed using 'rpm -qa' and how to uninstall it if necessary using 'rpm -e'. This is a crucial step before installing version 1.0.4 or later to avoid compatibility issues.

```bash
rpm -qa | grep -n validator
```

```bash
rpm -e ibm-aspera-otfv-validator
```

--------------------------------

### Create Node API User for Validator (Bash)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

This command creates a new Node API user on the HSTS, which the Validator will use for communication to retrieve file lists for validation. It requires a username, password, and association with a transfer user.

```bash
# asnodeadmin -a -u new_node_username -p new_node_password -x transfer_user
```

--------------------------------

### Schedule Validation Post-Transfer for Users, Groups, or Globally

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

Sets the validation timing to occur after a file transfer is complete. This can be configured for individual users, specific groups, or globally for all users and groups.

```shell
# asconfigurator -x "set_user_data;user_name,user_name;validation_file_stop,post_transfer"

```

```shell
# asconfigurator -x "set_group_data;group_name,group_name;validation_file_stop,post_transfer"

```

```shell
# asconfigurator -x "set_node_data;validation_file_stop,post_transfer"

```

--------------------------------

### Install Validator on CentOS

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-upgrading-validator

This command installs the new version of IBM Aspera Validator on CentOS systems using the RPM package manager. It ensures that the upgrade process overwrites the existing installation while preserving configuration files if needed.

```bash
# rpm -Uvh ibm-aspera-otfv-validator-version.rpm
```

--------------------------------

### Configure User and Group in Service File

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Example of how to specify the User and Group in the aspera-validator.service systemd file. These values should be updated to reflect the newly created system user and group.

```INI
User=<new-user>
Group=<new-group>
```

--------------------------------

### Get Package Environment Details in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Retrieves environment details for a given package ID. This function makes an HTTP request and may introduce a delay. It does not support Shared Inboxes or Work Groups. The output is a table containing package information.

```lua
local f = env["Path"]
local bytes_processed = env["Size"]

local penv = pkg_env(env.PackageId)

for i,v in pairs(penv) do
    print(i,v)
end

if f:match(".*failme.*") then
    local err_msg = "bad file name"
    update_status("error", bytes_processed, err_msg)
else
    update_status("completed", bytes_processed)
end
```

--------------------------------

### Configure HSTS to Not Ignore No Transfer Files

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-installing-validator-10

Configures the Host Transfer Server (HSTS) to not ignore files that do not transfer or are empty. Setting 'ignore_no_transfer_files' to 'false' ensures that these files are considered during validation, preventing potential workflow failures.

```shell
# asconfigurator -x "set_central_server_data;ignore_no_transfer_files,false"

```

--------------------------------

### De-obfuscate Passwords with Aspera Validator CLI

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=security-obfuscating-passwords-in-validator-102

This command reverses the obfuscation process, revealing plaintext passwords in the validator.yaml configuration file. It is used to revert changes made by the obfuscation command. Ensure you have Validator version 1.0.2 or later installed.

```bash
# /opt/aspera/aspera-validator/bin/aspera-validator deobfuscate
```

--------------------------------

### File System Operations with LuaFileSystem

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Utilizes the LuaFileSystem library to interact with the file system, enabling operations such as listing directory contents. Requires `require('lfs')` to load the library.

```lua
local lfs = require 'lfs'
for f in lfs.dir '.' do print(f) end
```

--------------------------------

### Change Directory Ownership

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Commands to change the ownership of the 'examples/' and 'config/' directories to 'root:otfv-group'. This is crucial for ensuring the service has the correct permissions.

```bash
chown -R root:otfv-group examples/
chown -R root:otfv-group config/
```

--------------------------------

### Configure Validator Servers

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Define the connection details for the servers that the Validator will perform validation against. This includes the server name, URL, and Node API authentication credentials. Ensure the URL is correctly formatted and credentials are valid.

```yaml
Servers:
- Name: node0
  URL: https://localhost:9092
  User: nodeuser
  Pass: nodepassword
```

--------------------------------

### MySQL Database Interaction in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Connects to and queries a MySQL database using the `mysql` library. Supports establishing a connection with credentials and executing SQL queries. Requires `require('mysql')`.

```lua
local mysql = require('mysql')
local c = mysql.new()
local ok, err = c:connect({ host = '127.0.0.1', port = 3306, database = 'test', user = 'user', password = 'pass' })
if ok then
  local res, err = c:query('SELECT * FROM mytable LIMIT 2')
  dump(res)
end
```

--------------------------------

### Logging Utility in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

A simple logging function that writes a given string to the `validator.log` file. It takes a single string argument.

```lua
aslog(string)
```

--------------------------------

### Restart Aspera Services

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-file-permissions-aspweb-user-when-using-clamav-anti-virus

These commands restart the 'aspera-validator' and 'asperanoded' services. Restarting is necessary to apply configuration changes, such as updated directory permissions.

```shell
systemctl restart asperanoded
systemctl restart aspera-validator
```

--------------------------------

### Network Requests with LuaSocket

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Employs the LuaSocket library for network communication, specifically for making HTTP requests. The `socket.http` module is used to fetch content from a given URL. Requires `require('socket')`.

```lua
local http = require("socket.http")
print(http.request("http://example.com"))
```

--------------------------------

### API Settings Configuration

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

This section details the configuration options available for the Aspera Validator API. These settings allow you to customize how the Validator interacts with servers, handles security, and manages its operational parameters.

```APIDOC
## API Settings Configuration

This API allows configuration of various settings for the Aspera Validator.

### Method
PUT

### Endpoint
/websites/ibm_en_aspera-validator_1_0_x/settings

### Parameters
#### Request Body
- **Servers** (array) - Required - List of servers for validation, each with Name, URL, User, and Pass.
- **RootCertificatePaths** (array) - Optional - Paths to root certificates for validation.
- **Insecure** (boolean) - Optional - Set to true to disable certificate validation. Default: false.
- **Proxy** (string) - Optional - URL for the proxy server.
- **FileQueueLimit** (integer) - Optional - Maximum number of files held in the queue. Default: 5.
- **UpdateBatchSize** (integer) - Optional - Number of messages cached before sending a batch to the server. Default: 10.
- **UpdateMaxInterval** (integer) - Optional - Duration to wait before sending a batch of cached messages. Default: 2.
- **FileWait** (integer) - Optional - Duration to wait before performing validation on a file. Default: 0.

### Request Example
```json
{
  "Servers": [
    {
      "Name": "node0",
      "URL": "https://localhost:9092",
      "User": "nodeuser",
      "Pass": "nodepassword"
    }
  ],
  "RootCertificatePaths": [
    "/opt/aspera/etc/aspera_server_cert.pem"
  ],
  "Insecure": false,
  "FileQueueLimit": 5,
  "UpdateBatchSize": 10,
  "UpdateMaxInterval": 2,
  "FileWait": 0
}
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message indicating settings have been updated.

#### Response Example
```json
{
  "message": "Validator settings updated successfully."
}
```
```

--------------------------------

### Aspera Validator Systemd Service Configuration

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Complete systemd service file for the Aspera Validator. This file defines the service's behavior, including execution commands, stop commands, and the user/group it runs as.

```INI
[Unit]
Description=start_aspera_validator_on_boot
After=syslog.target nss-lookup.target network.target

[Service]
ExecStart=/bin/bash -ce "/opt/aspera/aspera-validator/bin/aspera-validator"
ExecStop=/bin/bash -ce "kill -9 $(ps -Af | grep aspera-validator | grep -v grep | tr -s ' ' | cut -d" " -f2)"
TimeoutStartSec=0
TimeoutStopSec=0
User=<new-user>
Group=<new-group>

[Install]
WantedBy=multi-user.target
```

--------------------------------

### Change File Ownership

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Command to change the ownership of the 'version.txt' file to 'root:otfv-group'. This ensures the service can access and potentially modify this file if needed.

```bash
chown -R root:otfv-group version.txt
```

--------------------------------

### Configure Validator Log Settings (YAML)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=troubleshooting-validator-10-103-log-configuration

This snippet shows the YAML configuration structure for setting up logging in IBM Aspera Validator versions 1.0 to 1.0.3. It specifies parameters such as the log filename, enabling log rotation, maximum size in MB, number of files to retain, maximum age in days, compression settings, whether to use local time, and the desired log format (text or JSON).

```yaml
LogConfig:
  Filename: /var/log/validator.log
  EnableLogRotation: true
  MaxSizeMB: 100
  MaxFilesRetained: 10
  MaxAgeDays: 60
  Compress: true
  UseLocalTime: true
  Format: text # text or json
```

--------------------------------

### JSON Encoding and Decoding in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Integrates the JSON library to handle JSON data serialization and deserialization. Allows conversion of Lua tables to JSON strings and parsing JSON strings back into Lua tables. Requires `require('json')`.

```lua
local json = require("json")
local jstr = json.encode({example="yes"})
local data = json.decode(jstr)
print(data.example)
```

--------------------------------

### Validator Global Settings in validator.yaml

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Configures global settings for the IBM Aspera Validator. Options include enabling debug logging, specifying log file location and rotation policies, and setting a unique ValidatorID for reporting.

```yaml
Debug: false
LogConfig:
  Filename: /var/log/validator.log
  EnableLogRotation: true
  MaxSizeMB: 100
  MaxFilesRetained: 10
  MaxAgeDays: 60
  Compress: true
  UseLocalTime: true
  Format: text # text or json
ValidatorID: Not specified
```

--------------------------------

### Specify Root Certificate Paths

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Provide paths to alternative or additional root certificates for validating server SSL certificates. This is useful when using custom certificate authorities or when default paths do not contain the required certificates.

```yaml
RootCertificatePaths:
- /opt/aspera/etc/aspera_server_cert.pem
```

--------------------------------

### Check IBM Aspera Validator Service Status

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Checks the current status of the IBM Aspera Validator service using systemctl. This helps confirm if the service is active and running correctly after a restart or other operations.

```shell
systemctl status aspera-validator
```

--------------------------------

### Restart IBM Aspera Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Restarts the IBM Aspera Validator service using systemctl. This command is essential after making configuration changes to apply them.

```shell
systemctl restart aspera-validator
```

--------------------------------

### Uninstall Validator Package (Debian)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-uninstalling-validator

This command is used to uninstall the IBM Aspera Validator package on Debian-based systems. The specific package name might vary, but this represents a common approach. Ensure services are stopped prior to running this command.

```shell
# apt-get remove <package_name>
```

--------------------------------

### Update Package Status in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Functions to update the status of a package being processed. Supports marking as 'completed' or 'error', with an option to include a custom error message for failed packages. Requires package ID and bytes processed as input.

```lua
update_status(status, bytes_processed, err_msg)
```

```lua
update_status("completed", bytes_processed)
```

```lua
update_status("error", bytes_processed, "Custom error message")
```

--------------------------------

### URL Path Normalization in Lua

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=later-validator-104-configuration-reference

Removes the domain name and any extra query parameters from a URL path. This function helps in cleaning up URL strings for further processing.

```lua
url_path(path)
```

--------------------------------

### Validator Lua Settings in validator.yaml

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Configures settings related to Lua script execution within the IBM Aspera Validator. This includes setting the maximum number of concurrent validation threads (MaxRoutines) and the path to the Lua script to be executed on each transferred file.

```yaml
MaxRoutines: 8
LuaFilePath: /opt/aspera/aspera-validator/examples/basic.lua
```

--------------------------------

### Configure Proxy for Node API Access

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Specify a proxy URL for the Validator to use when accessing the Node API. This is necessary if the Validator is running in an environment that requires a proxy to reach external network resources.

```yaml
Proxy: http://your-proxy-url.com:8080
```

--------------------------------

### Reload Systemd Daemon

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Command to reload the systemd daemon after modifying service files. This ensures that systemd recognizes the changes made to the aspera-validator.service file.

```bash
systemctl daemon-reload
```

--------------------------------

### Configure File Wait Time

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Set the duration Validator waits before attempting to validate a file. This is recommended when using NFS async options to account for potential NFS caching delays, ensuring reliable file access and validation.

```yaml
FileWait: 0
```

--------------------------------

### Stop Aspera Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-running-validator-104-as-non-root-user

Command to stop the Aspera Validator service using systemctl. Ensure the service is stopped before making configuration changes.

```bash
systemctl stop aspera-validator
```

--------------------------------

### Change Directory Permissions for Faspex Packages

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=appendix-file-permissions-aspweb-user-when-using-clamav-anti-virus

This command changes the permissions of the Faspex packages directory to 755, allowing ClamAV to access it. It is a prerequisite for proper operation of the Aspera validator.

```shell
chmod 755 /home/faspex/
chmod -R 755 /home/faspex/faspex_packages
```

--------------------------------

### Uninstall Validator Package (CentOS)

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-uninstalling-validator

This command uninstalls the IBM Aspera Validator package on CentOS systems using the RPM package manager. Ensure Validator services are stopped before execution.

```shell
# rpm -e ibm-aspera-otfv-validator
```

--------------------------------

### Set File Queue Limit

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Define the maximum number of files that can be held in the Validator's queue. Once the queue reaches this limit, the Validator will pause requesting new files until space becomes available.

```yaml
FileQueueLimit: 5
```

--------------------------------

### Configure HSTS Validation Timeout with asconfigurator

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Sets the validation timeout for HSTS (HSTS considers a file in validation has timed out). This is configured using the asconfigurator command-line tool, not within the validator.yaml file. The default timeout is 5 minutes.

```bash
# asconfigurator -x "set_central_server_data;validation_timeout,value"
```

--------------------------------

### Set Insecure Certificate Validation

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Control whether certificate validation is disabled. Setting 'Insecure' to 'true' will disable certificate validation, which can be useful for testing environments but is not recommended for production due to security risks.

```yaml
Insecure: false
```

--------------------------------

### Stop Validator Service

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-uninstalling-validator

This command stops the IBM Aspera Validator service. It is a prerequisite before uninstalling the package. This command is typically run on Linux-based systems.

```shell
# service aspera-validator stop
```

--------------------------------

### Obfuscate Passwords with Aspera Validator CLI

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=security-obfuscating-passwords-in-validator-102

This command obfuscates plaintext passwords stored in the validator.yaml configuration file. It requires IBM Aspera Validator version 1.0.2 or later. The command modifies the 'Pass' field for server configurations.

```bash
# /opt/aspera/aspera-validator/bin/aspera-validator obfuscate
```

--------------------------------

### Configure Update Batch Size

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Determine the number of messages Validator caches before sending a batch to the server (asperacentral) for updating validation progress. A larger batch size can improve efficiency but may increase latency.

```yaml
UpdateBatchSize: 10
```

--------------------------------

### Set Update Maximum Interval

Source: https://www.ibm.com/docs/en/aspera-validator/1.0.x/index_topic=103-validator-10-configuration-reference

Specify the maximum duration Validator waits before sending a batch of cached messages to the server (asperacentral). This setting complements UpdateBatchSize to ensure timely progress updates even with small batches.

```yaml
UpdateMaxInterval: 2
```
Tokens:
Raw