### Initialize OpenSCOW Configuration Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/install Run this command to generate the main installation configuration file (`install.yaml`) and an example configuration directory (`config`). These files are essential for setting up OpenSCOW. ```bash ./cli init ``` -------------------------------- ### Install Golang Environment Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/cluster-monitor Steps to download, install, and configure the Go environment on the management node. ```bash # 下载go语言安装包,安装gocd download/ wget https://golang.google.cn/dl/go1.19.7.linux-amd64.tar.gz tar -C /usr/local -xzf go1.19.7.linux-amd64.tar.gz # 在/etc/profile中设置环境变量 export GOROOT=/usr/local/go export GOPATH=/usr/local/gopath export PATH=$PATH:/$GOROOT/bin:$GOPATH/bin # source环境变量 source /etc/profile # 验证 go version # 设置代理 go env -w GOPROXY=https://goproxy.cn,direct # 开启go mod管理 go env -w GO111MODULE=on ``` -------------------------------- ### Build and Start Local Development Environment Source: https://pkuhpc.github.io/OpenSCOW/docs/contribution/dev Commands to build and start the Docker-based development environment for local testing. Ensure 'prepareDev' has been run before starting tests. ```bash # Build and start the development environment pnpm devenv # Run tests (requires prepareDev to be completed first) pnpm test # Run tests for a specific project (e.g., @scow/portal-web) pnpm --filter portal-web test # Stop the development environment pnpm devenv:stop ``` -------------------------------- ### Install and Configure RNG Tools Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Install rng-tools to provide a source of entropy for Munge. Configure rngd to use /dev/urandom and ensure the service starts on boot. ```bash yum install -y rng-tools ``` ```bash rngd -r /dev/urandom ``` ```bash vim /usr/lib/systemd/system/rngd.service # Modify to: [service] ExecStart=/sbin/rngd -f -r /dev/urandom ``` ```bash systemctl daemon-reload systemctl start rngd systemctl enable rngd ``` -------------------------------- ### Install Desktop Environments Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/desktop Commands to install Xfce, KDE Plasma, MATE, and Cinnamon desktop environments using yum groupinstall or install. ```bash yum groupinstall "Xfce" -y yum groupinstall "KDE Plasma Workspaces" -y yum groupinstall "MATE Desktop" -y yum install cinnamon -y ``` -------------------------------- ### Clone Repository and Install Dependencies Source: https://pkuhpc.github.io/OpenSCOW/docs/contribution/dev Clone the OpenSCOW repository, navigate into the directory, and install project dependencies using pnpm. This includes installing node-gyp globally if necessary. ```bash git clone https://github.com/PKUHPC/OpenSCOW cd OpenSCOW # If using standalone installation or encountering node-gyp ENOET errors during pnpm i, run this global install. Only needs to be done once. pnpm install -g node-gyp # Install dependencies in the root directory pnpm i # Prepare development libraries and code: build dependencies, generate various code files pnpm prepareDev ``` -------------------------------- ### Install Modules Tool Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/module Download, compile, and install the modules tool, linking it against the previously installed Tcl library. ```bash mkdir -p /data/software/module/tools/modules ``` ```bash # 下载,此处可能需要科学上网 wget https://newcontinuum.dl.sourceforge.net/project/modules/Modules/modules-4.2.4/modules-4.2.4.tar.gz #解压 tar -zxvf modules-4.2.4.tar.gz cd modules-4.2.4 # 编译 & 安装,注意修改路径 ./configure --prefix=/data/software/module/tools/modules --with-tcl-lib=/data/software/module/tools/tcl/lib --with-tcl-inc=/data/software/module/tools/tcl/include make make install ``` -------------------------------- ### Minimal install.yaml Configuration Reference Source: https://pkuhpc.github.io/OpenSCOW/blog/tags/v-1-0-0 A minimal configuration example for install.yaml, showing essential parameters. ```yaml port: 80 basePath: / imageTag: v1.0.0 portal: portMappings: {} mis: dbPassword: must!chang3this portMappings: {} log: fluentd: logDir: /var/log/fluentd auth: portMappings: {} audit: dbPassword: "must!chang3this" gateway: proxyReadTimeout: 36000s ``` -------------------------------- ### Install MariaDB for Slurm Accounting Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Install and start the MariaDB server on the management node. Configure a root password and create the Slurm accounting database. ```bash yum -y install mariadb-server systemctl start mariadb systemctl enable mariadb ``` ```bash ROOT_PASS=$(tr -dc A-Za-z0-9 = 3.6.0 paramiko >= 3.0.0 psutil >= 5.9.4 subprocess concurrent ``` -------------------------------- ### Download and Install code-server Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/vscode Download the code-server binary, extract it, and move it to the specified software directory. Ensure you have the correct version. ```bash # 下载所需版本的code-server二进制安装包 wget https://github.com/coder/code-server/releases/download/v4.11.0/code-server-4.11.0-linux-amd64.tar.gz # 解压code-server二进制安装包 tar zxf code-server-4.11.0-linux-amd64.tar.gz # 将code-server重命名至指定路径 mkdir -p /data/software/code-server mv code-server-4.11.0-linux-amd64 /data/software/code-server/4.11.0 ``` -------------------------------- ### Set Up Slurm Directories and Permissions Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Create necessary spool and log directories for Slurm services and set the correct ownership to the 'slurm' user. ```bash mkdir /var/spool/slurmd chown slurm: /var/spool/slurmd mkdir /var/log/slurm chown slurm: /var/log/slurm mkdir /var/spool/slurmctld chown slurm: /var/spool/slurmctld ``` -------------------------------- ### Install TurboVNC Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/emacs Installs the TurboVNC software. Ensure you have the necessary repository configuration. ```bash wget https://turbovnc.org/pmwiki/uploads/Downloads/TurboVNC.repo --no-check-certificate mv TurboVNC.repo /etc/yum.repos.d # Install the latest version yum install turbovnc -y ``` -------------------------------- ### 配置安装文件 install.yaml Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/mis/intro 定义管理系统的部署路径和数据库初始密码。数据库密码在系统首次启动后不可修改。 ```yaml # 确保管理系统会部署 mis: # 如果管理系统部署在`/mis`路径下,设置"MIS.BASE_PATH"为 "/mis"。默认为"/" basePath: "/mis" # 如果将会部署在域名的根目录下,设置"MIS.BASE_PATH"为 "/" basePath: "/" # dbPassword为管理系统数据库密码 # 在系统第一次启动前可自由设置,使用此密码可以以root身份登录数据库 # 一旦数据库启动后即不可修改 # 必须长于8个字符,并同时包括字母、数字和符号 dbPassword: "must!chang3this" ``` -------------------------------- ### Install Tcl Dependency Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/module Download, compile, and install Tcl, which is a prerequisite for the module tool. ```bash mkdir -p /data/software/module/tools/tcl ``` ```bash # 下载 wget https://cfhcable.dl.sourceforge.net/project/tcl/Tcl/8.5.9/tcl8.5.9-src.tar.gz # 解压 tar -zxvf tcl8.5.9-src.tar.gz cd tcl8.5.9/unix # 编译 & 安装,注意修改此处目录 ./configure --prefix=/data/software/module/tools/tcl make make install ``` -------------------------------- ### 配置日志输出选项 Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/ops 在 install.yaml 中设置日志等级和输出格式。 ```yaml log: # LOG.LEVEL: 日志等级,可选trace, debug, info, warn, error。默认info level: "info" # LOG.PRETTY: 是否输出更适合直接读的log。如果为false,则输出json格式的日志。默认false pretty: false ``` -------------------------------- ### Install Emacs Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/emacs Installs the Emacs text editor using yum package manager. ```bash yum install emacs -y ``` -------------------------------- ### Start and Stop Services Source: https://pkuhpc.github.io/OpenSCOW/blog/tags/scow-deployment Commands to start or stop services using the compose script. ```bash # 启动服务 ./compose.sh up -d # 停止服务 ./compose.sh down ``` -------------------------------- ### 编写新版配置文件 Source: https://pkuhpc.github.io/OpenSCOW/blog/update-to-python-deployment 根据备份的.env文件内容修改新生成的config.py配置文件。 ```bash # 1. 复制配置文件 cp config-example.py config.py # 2. 配置参数 # 根据和备份的.env文件和config.py中的参数说明,修改config.py文件中的参数 ``` -------------------------------- ### Download and Prepare OpenSCOW CLI Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/install This snippet shows how to create a directory for OpenSCOW configuration, download the `openscow-cli` tool, and make it executable. Place the `scow-cli` binary in this directory. ```bash # scow目录将会用于存在OpenSCOW相关的配置文件 mkdir scow cd scow # 将下载的openscow-cli移动到scow目录下 cp /path/to/scow-cli ./ chmod +x scow-cli ``` -------------------------------- ### Prepare and Run Linkpack Benchmark Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/intel This section covers setting up the environment for the Intel MPI library, building the xhpl executable from the provided benchmarks, and running it locally and across multiple nodes. ```bash # 设置环境变量 source /data/software/intel/oneapi/mpi/2021.6.0/env/vars.sh # 生成xhpl cd /data/software/intel/oneapi/mkl/2022.1.0/benchmarks/mp_linpack ./build.sh # 执行计算 mpirun -n 36 ./xhpl # 多节点 mpirun -hosts cn1,cn2,cn3,cn4 -np 36 -ppn 4 /data/software/intel/oneapi/mkl/2022.1.0/benchmarks/mp_linpack/xhpl ``` -------------------------------- ### Install Jupyter Notebook Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/jupyter Command to install Jupyter Notebook if it is not already included with the Anaconda distribution. ```bash conda install jupyter notebook ``` -------------------------------- ### Install Slurm on All Nodes Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Install the Slurm packages on all nodes using yum localinstall after copying the RPMs. ```bash cd /root/rpmbuild/RPMS/x86_64/ yum localinstall slurm-* ``` -------------------------------- ### 配置安装数据库密码 Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/ai/intro 在 install.yaml 中设置 AI 系统数据库的初始 root 密码。 ```yaml # 其他配置省略 # ... # 确保 AI 系统会部署 ai: # dbPassword 为 AI 系统数据库密码 # 在系统第一次启动前可自由设置,使用此密码可以以 root 身份登录数据库 # 一旦数据库启动后即不可修改 # 必须长于 8 个字符,并同时包括字母、数字和符号 dbPassword: "must!chang3this" ``` -------------------------------- ### Install JupyterLab via Conda Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/jupyterlab Use this command to install JupyterLab if it was not automatically included with your Anaconda distribution. ```bash conda install jupyterlab ``` -------------------------------- ### Install Octave via Conda Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/octave Command to install the specific version of Octave within the active environment. ```bash conda install octave=7.2.0 ``` -------------------------------- ### Minimal mis.yaml Configuration Reference Source: https://pkuhpc.github.io/OpenSCOW/blog/tags/v-1-0-0 A minimal configuration example for mis.yaml, including database settings and job fetching. ```yaml db: host: db port: 3306 user: root dbName: scow fetchJobs: periodicFetch: enabled: true cron: "*/10 * * * *" predefinedChargingTypes: - 测试 accountNamePattern: regex: "(a_)[a-z]+$" ``` -------------------------------- ### Install Java 1.8 Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/igv Installs OpenJDK version 1.8 using yum. This is a prerequisite for running IGV. ```bash yum install java-1.8.0-openjdk ``` -------------------------------- ### Install TurboVNC Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/desktop Commands to download the TurboVNC repository configuration and install the latest version via yum. ```bash wget https://turbovnc.org/pmwiki/uploads/Downloads/TurboVNC.repo --no-check-certificate mv TurboVNC.repo /etc/yum.repos.d # 安装最新版本 yum install turbovnc -y ``` -------------------------------- ### Initialize Database Schema Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/cluster-monitor SQL script to create the initial database tables and a bootstrap procedure for version tracking. ```sql DROP PROCEDURE IF EXISTS bootstrap; DELIMITER // CREATE PROCEDURE bootstrap() BEGIN SET @exists := (SELECT 1 FROM information_schema.tables I WHERE I.table_name = "Model" AND I.table_schema = database()); IF @exists IS NULL THEN CREATE TABLE `Model` ( `ID` enum('1') NOT NULL, `version` VARCHAR(20) NOT NULL, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; INSERT INTO `Model` (`version`) VALUES ("0.0.1"); ELSE SIGNAL SQLSTATE '42000' SET MESSAGE_TEXT='Model Table Exists, quitting...'; END IF; END; // DELIMITER ; -- Execute the procedure CALL bootstrap(); -- Drop the procedure DROP PROCEDURE bootstrap; -- Create the rest of the tables CREATE TABLE `AlertGroup` ( `ID` INT NOT NULL AUTO_INCREMENT, `time` TIMESTAMP NOT NULL, `receiver` VARCHAR(100) NOT NULL, `status` VARCHAR(50) NOT NULL, `externalURL` TEXT NOT NULL, `groupKey` VARCHAR(255) NOT NULL, KEY `idx_time` (`time`) USING BTREE, KEY `idx_status_ts` (`status`, `time`) USING BTREE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `GroupLabel` ( `ID` INT NOT NULL AUTO_INCREMENT, `AlertGroupID` INT NOT NULL, `GroupLabel` VARCHAR(100) NOT NULL, `Value` VARCHAR(1000) NOT NULL, FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `CommonLabel` ( `ID` INT NOT NULL AUTO_INCREMENT, `AlertGroupID` INT NOT NULL, `Label` VARCHAR(100) NOT NULL, `Value` VARCHAR(1000) NOT NULL, FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `CommonAnnotation` ( `ID` INT NOT NULL AUTO_INCREMENT, `AlertGroupID` INT NOT NULL, `Annotation` VARCHAR(100) NOT NULL, `Value` VARCHAR(1000) NOT NULL, FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `Alert` ( `ID` INT NOT NULL AUTO_INCREMENT, `alertGroupID` INT NOT NULL, `status` VARCHAR(50) NOT NULL, `startsAt` DATETIME NOT NULL, `endsAt` DATETIME DEFAULT NULL, `generatorURL` TEXT NOT NULL, FOREIGN KEY (alertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `AlertLabel` ( `ID` INT NOT NULL AUTO_INCREMENT, `AlertID` INT NOT NULL, `Label` VARCHAR(100) NOT NULL, `Value` VARCHAR(1000) NOT NULL, FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE `AlertAnnotation` ( `ID` INT NOT NULL AUTO_INCREMENT, `AlertID` INT NOT NULL, `Annotation` VARCHAR(100) NOT NULL, `Value` VARCHAR(1000) NOT NULL, FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE, PRIMARY KEY (`ID`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; ``` -------------------------------- ### User Dropdown Link Configuration Examples Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/customization/custom-userlinks Demonstrates different ways to configure user links, including external URLs, internal paths, and how they are handled during navigation. The system appends a token for tracking user state. ```yaml userLinks: - text: "菜单1" url: "http://www.navtest1.com" - text: "菜单2" url: "www.navtest2.com" - text: "菜单3" url: "/www.navtest3.com" ``` -------------------------------- ### Install Slurm Dependencies Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Install necessary development tools and libraries required for compiling and running Slurm on all nodes. ```bash yum install gcc gcc-c++ readline-devel perl-ExtUtils-MakeMaker pam-devel rpm-build mysql-devel python3 -y ``` -------------------------------- ### Install R via Conda Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/rstudio Use Conda to create a virtual environment and install a specific version of R. ```bash # 载入conda到系统环境中,并初始化conda环境 module load anaconda/3-2023.03 eval "$($(which conda) shell.bash hook)" # 添加北大镜像源,查找相应版本的R进行安装 conda config --add channels https://mirrors.pku.edu.cn/anaconda/pkgs/free/ conda config --add channels https://mirrors.pku.edu.cn/anaconda/pkgs/main/ conda config --add channels https://mirrors.pku.edu.cn/anaconda/cloud/conda-forge/ conda search R # 根据查找到的R版本,创建Conda虚拟环境,并在创建的时候指定需要安装的R版本,这里选择安装R-3.6.0版本; conda create -n R-3.6.0 r=3.6.0 -y # 进入上一步创建好的虚拟环境; conda activate R-3.6.0 # 安装R语言库,比如stringi conda install r-stringi -y # 安装完后就可以用R语言环境了,如果要退出当前环境则执行以下指令; conda deactivate ``` -------------------------------- ### Run Portal Frontend in Development Mode Source: https://pkuhpc.github.io/OpenSCOW/docs/contribution/dev Navigate to the portal frontend directory and run the 'dev' script using pnpm to start the application in development mode. ```bash cd apps/portal-web pnpm dev ``` -------------------------------- ### Install TurboVNC Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/config/portal/apps/apps/igv Installs the latest version of TurboVNC using yum. Ensure you have the TurboVNC repository file configured. ```bash wget https://turbovnc.org/pmwiki/uploads/Downloads/TurboVNC.repo --no-check-certificate mv TurboVNC.repo /etc/yum.repos.d # 安装最新版本 yum install turbovnc -y ``` -------------------------------- ### Minimal install.yaml Configuration Source: https://pkuhpc.github.io/OpenSCOW/blog/tags/scow A basic configuration reference for `install.yaml`, including port, basePath, imageTag, and settings for portal, mis, log, auth, audit, and gateway. ```yaml port: 80 basePath: / imageTag: v1.0.0 portal: portMappings: {} mis: dbPassword: must!chang3this portMappings: {} log: fluentd: logDir: /var/log/fluentd auth: portMappings: {} audit: dbPassword: "must!chang3this" gateway: proxyReadTimeout: 36000s ``` -------------------------------- ### Install Munge Authentication Service Source: https://pkuhpc.github.io/OpenSCOW/docs/slurm Install Munge and its development libraries on all nodes. Munge is essential for authenticating processes between local or remote hosts. ```bash yum install epel-release -y yum install munge munge-libs munge-devel -y ``` -------------------------------- ### Start Monitoring Services with Docker Compose Source: https://pkuhpc.github.io/OpenSCOW/docs/hpccluster/cluster-monitor Command to start the monitoring services defined in the docker-compose.yaml file. This should be run after the configuration file is edited. ```bash cd /root/monitor docker-compose -f monitor.yaml up -d ``` -------------------------------- ### Run OpenSCOW API Client Source: https://pkuhpc.github.io/OpenSCOW/docs/integration/scow-api-hook/examples/go Download Go module dependencies and run the OpenSCOW API client code. Ensure the Go environment is set up correctly. ```bash go mod tidy go run api.go ``` -------------------------------- ### Start and Stop Services Source: https://pkuhpc.github.io/OpenSCOW/blog Manages the OpenCow services using Docker Compose. Use 'up -d' to start services in detached mode and 'down' to stop them. ```bash # 启动服务 ./compose.sh up -d # 停止服务 ./compose.sh down ``` -------------------------------- ### Login/Compute节点镜像配置 Source: https://pkuhpc.github.io/OpenSCOW/docs/deploy/get-started/vagrant/images 执行此脚本以配置基础系统环境、安装桌面环境、TurboVNC、Munge及Slurm。 ```bash # 1. 升级到centos7.9.2009 yum update -y # 2. 关闭firewalld、dnsmasq、NetworkManager systemctl disable --now firewalld systemctl disable --now dnsmasq systemctl disable --now NetworkManager # 3. 关闭selinux setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config getenforce # 4. 关闭swap swapoff -a && sysctl -w vm.swappiness=0enforcing sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab # 5. 时间同步 rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm yum install ntpdate vim -y # 同步时间。time2.aliyun.com外网,vineyard.pku.edu.cn内网 # 时间同步配置如下: ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime echo 'Asia/Shanghai' >/etc/timezone ntpdate time2.aliyun.com cat >>/var/spool/cron/root <