### Run XDMoD Setup Wizard Source: https://context7.com/ubccr/xdmod/llms.txt Use `xdmod-setup` to interactively configure your Open XDMoD installation, including database settings, site information, and admin user creation. ```bash # Run the setup wizard xdmod-setup # Configuration includes: # - Site address (URL for accessing Open XDMoD) # - Email address for notifications # - Database hostname, port, username, password # - Organization name and abbreviation # - Resource definitions (clusters) # - Admin user creation # - Hierarchy configuration ``` -------------------------------- ### Run XDMoD Setup Source: https://github.com/ubccr/xdmod/blob/main/docs/install-rpm.md Execute the setup command after installing the RPM package. Ensure MySQL is running beforehand. ```bash # xdmod-setup ``` -------------------------------- ### Run Open XDMoD Setup Script Source: https://github.com/ubccr/xdmod/blob/main/docs/install-source.md Execute the primary setup script for Open XDMoD. This script will guide you through configuring various aspects of the installation. ```bash # /opt/xdmod-{{ page.sw_version }}/bin/xdmod-setup ``` -------------------------------- ### Example authsources.php Configuration Source: https://github.com/ubccr/xdmod/blob/main/docs/simpleSAMLphp.md Complete example of an authsources.php file with attribute mapping and authorization checks. ```php array( 'saml:SP', // 'idp' => 'urn:example:idp', 'signature.algorithm' => 'http://www.w3.org/2001/04/xmldsig-more#rsa-sha256', 'authproc' => array( 40 => array( 'class' => 'core:AttributeMap', 'email' => 'email_address', 'firstName' => 'first_name', 'middleName' => 'middle_name', 'lastName' => 'last_name', 'personId' => 'person_id', 'orgId' => 'organization', 'fieldOfScience' => 'field_of_science', 'itname' => 'username' ), // Ensures that the 'username' property has one or more non-whitespace characters 60 => array( 'class' => 'authorize:Authorize', 'username' => array( '/\S+/' ), ), // Ensures that the 'organization' property has one or more non-whitespace characters 61 => array( 'class' => 'authorize:Authorize', 'organization' => array( '/\S+/' ) ) ) ), ); ``` -------------------------------- ### Install EPEL and Required Packages Source: https://github.com/ubccr/xdmod/blob/main/docs/software-requirements.md Install the EPEL repository and necessary system packages for Open XDMoD. ```sh dnf install -y epel-release dnf install -y php make libzip-devel php-pear php-devel \ mariadb-server mariadb ``` -------------------------------- ### Install Oracle Libraries and OCI8 Source: https://github.com/ubccr/xdmod/blob/main/docs/etl/data-endpoint-oracle.md Commands to install Oracle Instant Client and the PHP OCI8 PECL extension on Linux. ```bash sudo -s cd /opt unzip instantclient-basic-linux.x64-12.1.0.2.0.zip unzip instantclient-sdk-linux.x64-12.1.0.2.0.zip ln -s instantclient_12_1 oracle chmod -R g+rX,o+rX instantclient_12_1 cd instantclient_12_1 ln -s libclntsh.so.12.1 libclntsh.so ln -s libocci.so.12.1 libocci.so pecl install oci8 ``` -------------------------------- ### Command-line execution examples Source: https://github.com/ubccr/xdmod/blob/main/docs/docs-conventions.md Examples of command-line usage for unprivileged and root users, including path adjustments and placeholder values. ```bash $ xdmod-ingestor -v ``` ```bash # ./install --prefix=/opt/xdmod-{{ page.sw_version }} ``` ```bash # xdmod-setup ``` ```bash # /opt/xdmod-{{ page.sw_version }}/bin/xdmod-setup ``` ```bash $ xdmod-shredder -f *format* -r *resource* -i *input* ``` ```bash $ xdmod-shredder -f sge ... ``` -------------------------------- ### MySQL statement examples Source: https://github.com/ubccr/xdmod/blob/main/docs/docs-conventions.md Examples of MySQL queries and database selection statements. ```sql mysql> SELECT * FROM Users; ``` ```sql mysql> use moddb; mysql> SELECT * FROM Users; ``` ```sql mysql> SELECT * FROM moddb.Users; ``` -------------------------------- ### Install RPM Package Manually Source: https://github.com/ubccr/xdmod/blob/main/docs/install-rpm.md Download the RPM file from GitHub and install it manually if direct HTTPS installation is not possible. ```bash # dnf install xdmod-{{ page.rpm_version }}.el8.noarch.rpm ``` -------------------------------- ### Install RPM Package Directly Source: https://github.com/ubccr/xdmod/blob/main/docs/install-rpm.md Use this command to install the RPM package if your web server can reach GitHub via HTTPS. ```bash # dnf install https://github.com/ubccr/xdmod/releases/download/v{{ page.rpm_version }}/xdmod-{{ page.rpm_version }}.el8.noarch.rpm ``` -------------------------------- ### Install Node.js Dependencies Source: https://github.com/ubccr/xdmod/blob/main/etl/js/README.txt Run this command in the xdmod/etl/js directory to install necessary Node.js package dependencies. ```bash npm install ``` -------------------------------- ### Create SimpleSAMLphp Directories Source: https://github.com/ubccr/xdmod/blob/main/docs/simpleSAMLphp.md Create the configuration and metadata directories for SimpleSAMLphp. Ensure these directories exist before proceeding with the SSO setup. ```bash # mkdir -p /etc/xdmod/simplesamlphp/config # mkdir -p /etc/xdmod/simplesamlphp/metadata ``` -------------------------------- ### PBS Resource_List.ngpus Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of the Resource_List.ngpus field in PBS accounting logs, directly specifying the total number of GPUs. ```text Resource_List.ngpus=4 ``` -------------------------------- ### Add IDP Metadata to XDMoD SP Source: https://github.com/ubccr/xdmod/blob/main/docs/simpleSAMLphp-ldap.md Copy the IDP's metadata from the SimpleSAMLphp installation page into the XDMoD SP's remote metadata configuration file. Ensure the file starts with ` 'saml20-idp-remote', 'entityid' => 'xdmod-hosted-idp-ldap', 'SingleSignOnService' => array ( 0 => array ( 'Binding' => 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect', 'Location' => 'https://{% raw %}{{HOSTNAME}}{% endraw %}/simplesaml/saml2/idp/SSOService.php', ), ), 'SingleLogoutService' => array ( 0 => array ( 'Binding' => 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect', 'Location' => 'https://{% raw %}{{HOSTNAME}}{% endraw %}/simplesaml/saml2/idp/SingleLogoutService.php', ), ), 'certData' => '...', 'NameIDFormat' => 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient', ); ``` -------------------------------- ### PBS Resource_List.nodes Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of the Resource_List.nodes field in PBS accounting logs, specifying nodes and GPUs per node. ```text Resource_List.nodes=2:ppn=32:gpus=2 ``` -------------------------------- ### Install Composer Dependencies Source: https://github.com/ubccr/xdmod/blob/main/README.md Run this command in the root of the Open XDMoD repository to install required dependencies. ```bash composer install ``` -------------------------------- ### Install Node.js 16 on Rocky 8 Source: https://github.com/ubccr/xdmod/blob/main/docs/software-requirements.md Reset and install the Node.js 16 module stream. ```shell dnf module -y reset nodejs dnf module -y install nodejs:16 ``` -------------------------------- ### Install Open XDMoD Source Package Source: https://github.com/ubccr/xdmod/blob/main/docs/install-source.md Extract the source package and install Open XDMoD to a specified prefix. Ensure you download the correct tarball from GitHub releases. ```bash # tar zxvf xdmod-{{ page.sw_version }}.tar.gz # cd xdmod-{{ page.sw_version }} # ./install --prefix=/opt/xdmod-{{ page.sw_version }} ``` -------------------------------- ### Non-Standard PBS Resource_List.gpu Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of a non-standard PBS Resource_List.gpu field, used to determine GPU count when other methods are unavailable. ```text Resource_List.gpu=2 ``` -------------------------------- ### Slurm AllocTRES Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of the AllocTRES accounting field from sacct output, indicating allocated resources including GPUs. ```text billing=10,cpu=10,gres/gpu=4,mem=374000M,node=2 ``` -------------------------------- ### Copy Apache Configuration File Source: https://github.com/ubccr/xdmod/blob/main/docs/install-source.md Copy the example Apache configuration file to the appropriate directory for your web server. This file may require further editing for site-specific settings. ```bash # cp /opt/xdmod-{{ page.sw_version }}/share/templates/apache.conf /etc/apache2/conf.d/xdmod.conf ``` -------------------------------- ### Non-Standard PBS Resource_List.nodect Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of a non-standard PBS Resource_List.nodect field, indicating GPUs per node when standard methods fail. ```text Resource_List.nodect=2:ppn=32:gpu=2 ``` -------------------------------- ### Install PHP MongoDB Driver via PECL Source: https://github.com/ubccr/xdmod/blob/main/docs/software-requirements.md Install the MongoDB driver using PECL and configure the PHP extension. ```shell pecl install mongodb-1.18.1 echo "extension=mongodb.so" > /etc/php.d/40-mongodb.ini ``` -------------------------------- ### JSON Reference Examples Source: https://github.com/ubccr/xdmod/blob/main/docs/etl/json-references.md Illustrates different ways to use JSON references to include external or internal data. These examples show how to reference entire documents, specific parts of documents, or parts of documents in external files. ```json { "$ref": "table.json" } ``` ```json { "$ref": "table.json#/foo/0" } ``` ```json { "$ref": "#/foo" } ``` ```json { "$ref": "http://example.com/example.json#/foo" } ``` ```json { "$ref": "example.json#/foo/bar" } ``` ```json { "$ref": "file:///path/to/example.json#/foo/bar" } ``` -------------------------------- ### Install Open XDMoD RPM Package Source: https://github.com/ubccr/xdmod/blob/main/docs/upgrade.md Commands to install the Open XDMoD RPM package either directly from GitHub or from a local file. ```bash # dnf install https://github.com/ubccr/xdmod/releases/download/v{{ page.rpm_version }}/xdmod-{{ page.rpm_version }}.el8.noarch.rpm [optional module RPMs] ``` ```bash # dnf install xdmod-{{ page.rpm_version }}.el8.noarch.rpm [optional module RPMs] ``` -------------------------------- ### Copy Current Config Files Source: https://github.com/ubccr/xdmod/blob/main/docs/upgrade.md Commands to migrate existing configuration files from the previous version directory to the new installation directory. ```bash # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/portal_settings.ini /opt/xdmod-{{ page.sw_version }}/etc # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/hierarchy.json /opt/xdmod-{{ page.sw_version }}/etc # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/organization.json /opt/xdmod-{{ page.sw_version }}/etc # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/resource_specs.json /opt/xdmod-{{ page.sw_version }}/etc # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/resources.json /opt/xdmod-{{ page.sw_version }}/etc # cp /opt/xdmod-{{ page.prev_sw_version }}/etc/update_check.json /opt/xdmod-{{ page.sw_version }}/etc ``` -------------------------------- ### Configure Local Naming Method Source: https://github.com/ubccr/xdmod/blob/main/docs/etl/data-endpoint-oracle.md Configuration for tnsnames.ora and environment variable setup for local naming. ```text datawarehouse = (description = (address = (protocol = tcp)(host = db.mycompany.com)(port = 1521)) (connect_data = (service_name = mydb))) ``` ```bash export TNS_ADMIN=~/oracle/tns ``` -------------------------------- ### Example Dashboard Component JavaScript Class Source: https://github.com/ubccr/xdmod/blob/main/docs/dashboard-devel.md This JavaScript class extends Ext.ux.Portlet and serves as a template for new dashboard components. It includes example event handling for 'duration_change' and 'refresh'. Ensure the class is placed in the conventional directory and registered with Ext.reg. ```javascript /** * XDMoD.Module.Dashboard.DeveloperComponent * * This is an example panel that can be used as a template. */ Ext.namespace('XDMoD.Module.Dashboard'); XDMoD.Module.Dashboard.DeveloperComponent = Ext.extend(Ext.ux.Portlet, { layout: 'fit', autoScroll: true, title: 'XDMoD software developer info', bbar: { items: [{ text: 'Reset Layout', handler: function () { Ext.Ajax.request({ url: XDMoD.REST.url + '/summary/layout', method: 'DELETE' }); } }] }, /** * */ initComponent: function () { var aspectRatio = 0.8; this.height = this.width * aspectRatio; this.items = [{ itemId: 'console', html: '
' + JSON.stringify(this.config, null, 4) + '
' }]; XDMoD.Module.Dashboard.DeveloperComponent.superclass.initComponent.apply(this, arguments); }, listeners: { /** * duration_change event gets fired when the duration settings * in the top toolbar are changed by the user or when the refresh * button is clicked. * * A typical component will reload its content with the updated * duration parameters. */ duration_change: function (timeframe) { var console = this.getComponent('console'); console.body.insertHtml('beforeBegin', '
' + JSON.stringify(timeframe, null, 4) + '
'); }, /** * the component should refresh itself if the refresh event fires */ refresh: function () { var console = this.getComponent('console'); console.body.insertHtml('beforeBegin', '
refresh event fired
'); } } }); /** * The Ext.reg call is used to register an xtype for this class so it * can be dynamically instantiated */ Ext.reg('xdmod-dash-developer-cmp', XDMoD.Modules.Dashboard.DeveloperComponent); ``` -------------------------------- ### Define PHPUnit Test Suite Source: https://github.com/ubccr/xdmod/blob/main/tests/component/README.md Example configuration for a test suite within the phpunit.xml.dist file. ```xml lib lib/SlurmHelperTest.php ``` -------------------------------- ### Comment Transformer JSON Example Source: https://github.com/ubccr/xdmod/blob/main/docs/etl/json-references.md Example of a JSON structure containing a comment key starting with #, which is removed by the transformer. ```json { "# comment": { "name": "Joe" } } ``` -------------------------------- ### Sync Repositories with Repo Source: https://github.com/ubccr/xdmod/blob/main/README.md Download all repositories defined in the Repo manifest. This command should be run after `repo init`. ```bash repo sync ``` -------------------------------- ### XDMOD ETL Batch Aggregation Output Log Source: https://github.com/ubccr/xdmod/wiki/Aggregation_Scalability Example log output from the XDMOD ETL overseer script during batch aggregation. This shows the setup time for a batch of 20 periods and the aggregation progress for individual days within that batch. ```log 2016-11-08 13:05:25 [info] Setup for batch (day_id 201600286 - 201600305): 20.97s 2016-11-08 13:05:26 [info] Aggregated day (1 of 5041) 201600305 records = 2116, time = 1.06s 2016-11-08 13:05:27 [info] Aggregated day (2 of 5041) 201600304 records = 1382, time = 1.14s ... 2016-11-08 13:05:43 [info] Total time for batch (day_id 201600286 - 201600305): 39.1s ``` -------------------------------- ### Initialize Repo with Custom Manifest Source: https://github.com/ubccr/xdmod/blob/main/README.md Initialize Repo using a custom manifest repository, either local or remote. This allows for managing custom module configurations. ```bash repo init -u [local_or_remote_path_to_manifest_repo] -b [branch_with_changes] ``` -------------------------------- ### List configured resources Source: https://github.com/ubccr/xdmod/blob/main/docs/commands.md Lists all resources configured in the resources.json file. ```bash xdmod-admin --list --resources ``` -------------------------------- ### Initialize Repo for Open XDMoD Source: https://github.com/ubccr/xdmod/blob/main/README.md Initialize the Repo client to manage Open XDMoD and its first-party modules. Specify the desired branch for your work. ```bash repo init -u git@github.com:ubccr/xdmod-repo-manifest -b [branch] ``` -------------------------------- ### Configure Portal Settings Source: https://context7.com/ubccr/xdmod/llms.txt INI configuration for portal, database, and export settings in portal_settings.ini. ```ini [general] site_address = "https://xdmod.example.org" user_manual = "https://open.xdmod.org" tech_support_recipient = "hpc-support@example.org" maintainer_email_signature = "HPC Support Team" center_logo = "logo.png" center_logo_width = 200 chromium_path = "/usr/bin/chromium-browser" [database] host = "localhost" port = "3306" user = "xdmod" pass = "password" [slurm] sacct = "/usr/bin/sacct" [data_warehouse_export] export_directory = "/var/spool/xdmod/export" retention_duration_days = 30 hash_salt = "random_salt_value" ``` -------------------------------- ### Display help for xdmod-slurm-helper Source: https://github.com/ubccr/xdmod/blob/main/docs/resource-manager-slurm.md Execute this command to view the help text and available options for the `xdmod-slurm-helper` script. ```bash xdmod-slurm-helper -h ``` -------------------------------- ### Configure SimpleSAMLphp Paths Source: https://github.com/ubccr/xdmod/blob/main/docs/simpleSAMLphp.md Snippet for setting the certificate directory and metadata sources in config.php. ```php ... 'certdir' => '/etc/xdmod/simplesamlphp/cert/', 'metadata.sources' => array( array( 'type' => 'flatfile', 'directory' => '/etc/xdmod/simplesamlphp/metadata/' ), ), ... ``` -------------------------------- ### Create SimpleSAMLphp Certificate Directory Source: https://github.com/ubccr/xdmod/blob/main/docs/simpleSAMLphp.md Command to create the directory for SimpleSAMLphp certificates. ```bash mkdir -p /etc/xdmod/simplesamlphp/cert ``` -------------------------------- ### Build Script Usage Source: https://github.com/ubccr/xdmod/blob/main/user_manual_builder/README.md Usage instructions for the manual build scripts. ```shell setup_user_manual.sh -b|--builddir : Directory where manuals are built from -d|--destdir : Directory where the built manual is stored -v|--version : Manual version to be linked (Open or XSEDE) ``` ```shell build_user_manual.sh -b|--builddir : Directory where manuals are built from -d|--destdir : Directory where the built manual is stored -v|--version : Manual version to be linked (Open or XSEDE) ``` -------------------------------- ### Grid Engine (UGE) Category Field Example Source: https://github.com/ubccr/xdmod/blob/main/docs/gpu-metrics.md Example of the category field in Univa Grid Engine accounting logs, specifying the number of GPUs allocated to a job. ```text -l gpu=1 ``` -------------------------------- ### Bootstrap the gateways realm Source: https://github.com/ubccr/xdmod/blob/main/docs/gateways-realm.md Creates the modw_gateways database and necessary tables. ```bash $ /usr/share/xdmod/tools/etl/etl_overseer.php -p gateways.bootstrap ``` -------------------------------- ### Example Storage Metrics JSON Input Source: https://github.com/ubccr/xdmod/blob/main/docs/storage.md This is an example of the JSON format required for storage metrics data. Ensure all values are in bytes and dates are in RFC 3339 UTC format. ```json [ { "resource": "nfs", "mountpoint": "/home", "user": "jdoe", "pi": "pi_username", "dt": "2017-01-01T00:00:00Z", "soft_threshold": 1000000, "hard_threshold": 1200000, "file_count": 10000, "logical_usage": 100000, "physical_usage": 100000 }, ... ] ``` -------------------------------- ### Preconfigure SSO user accounts Source: https://github.com/ubccr/xdmod/blob/main/docs/commands.md Loads user settings from a CSV file to preconfigure SSO accounts. ```bash xdmod-admin --users --load PATH/TO/USERSFILE.csv ``` ```bash xdmod-admin --users --load FILENAME.csv ``` -------------------------------- ### GET /rest/metrics/explorer/queries Source: https://context7.com/ubccr/xdmod/llms.txt Retrieves saved chart configurations. ```APIDOC ## GET /rest/metrics/explorer/queries ### Description Get all saved charts. ### Method GET ### Endpoint https://xdmod.example.org/rest/metrics/explorer/queries ``` -------------------------------- ### Run Ingestor with Last Modified Start Date Source: https://github.com/ubccr/xdmod/blob/main/docs/ingestor.md Specifies the earliest date for jobs to be included in aggregation. Defaults to the start of the ingest process. Date format must be ISO 8601 (e.g., "2019-01-01" or "2019-01-01 12:00:00"). ```bash $ xdmod-ingestor --last-modified-start-date *date* ``` -------------------------------- ### Enable InnoDB File Per Table Source: https://github.com/ubccr/xdmod/blob/main/docs/configuration.md Configuration line to add to the /etc/my.cnf file. ```ini innodb_file_per_table = On ``` -------------------------------- ### GET /ubccr/xdmod/resources.json Source: https://github.com/ubccr/xdmod/blob/main/docs/configuration.md Retrieves the current configuration for all defined resources. ```APIDOC ## GET /ubccr/xdmod/resources.json ### Description Retrieves a list of all configured resources, including their properties and settings. ### Method GET ### Endpoint /ubccr/xdmod/resources.json ### Parameters None ### Request Example None ### Response #### Success Response (200) - **Array of Resource Objects** (object[]) - A list of resource configuration objects. #### Response Example ```json [ { "resource": "resource1", "name": "Resource 1", "description": "Our first HPC resource", "resource_type": "HPC", "resource_allocation_type": "CPUNode" }, { "resource": "resource2", "name": "Resource 2", "resource_type": "HPC", "pi_column": "account_name", "resource_allocation_type": "GPU" } ] ``` ``` -------------------------------- ### Verify MongoDB Driver Installation Source: https://github.com/ubccr/xdmod/blob/main/docs/software-requirements.md Check if the MongoDB extension is loaded in PHP. ```shell php -i | grep mongo ``` -------------------------------- ### Figure Directive Usage Source: https://github.com/ubccr/xdmod/blob/main/user_manual_builder/README.md Example of using the figure directive with a name and caption. ```rst .. figure:: path/to/image :name: name_of_image caption text ``` -------------------------------- ### Data Input Formats Source: https://context7.com/ubccr/xdmod/llms.txt Examples of data input formats accepted by the system. ```APIDOC ## Storage Metrics JSON ### Description Format for providing storage metrics data. ### Format JSON Array ### Example ```json [ { "resource": "nfs", "mountpoint": "/home", "user": "jdoe", "pi": "pi_username", "dt": "2024-01-01T00:00:00Z", "soft_threshold": 107374182400, "hard_threshold": 214748364800, "file_count": 150000, "logical_usage": 85899345920, "physical_usage": 85899345920 } ] ``` ``` -------------------------------- ### reStructuredText Syntax Example Source: https://github.com/ubccr/xdmod/blob/main/user_manual_builder/README.md Basic syntax for headings, lists, and inline formatting in reStructuredText. ```rst =========== Heading 1 =========== Heading 2 --------- Heading 3 ~~~~~~~~~ - Bullet point 1 - Bullet point 2 1. Numbered item 1 2. Numbered item 2 **Bold text** *Italic text* `Link `_ ``` -------------------------------- ### PHP 5.3: Initial Array Values Source: https://github.com/ubccr/xdmod/wiki/PHP_notes Demonstrates the initial values of an array in PHP 5.3 before filtering. ```php $a = array(0, 1, true, false, "true", "false", "totalcrap"); foreach ($a as $v) { print var_dump($v); } ```