### Start and Stop WeFe Service Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md Commands to start and stop the WeFe system service using the provided `wefe-service.sh` shell script. ```shell # 启动 sh wefe-service.sh start # 停止 sh wefe-service.sh stop ``` -------------------------------- ### Compile pybind11 C++ Example on Linux/macOS Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This bash command compiles the `example.cpp` file into a shared library for Python on Linux and macOS. It uses `c++` with optimization, warning flags, C++11 standard, and includes pybind11 headers. The output is a Python-loadable module. ```bash $ c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example$(python3-config --extension-suffix) ``` -------------------------------- ### Compile pybind11 Test Cases on Linux/macOS Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This snippet provides commands to compile and run pybind11 test cases on Linux and macOS. It requires `python-dev` (or `python3-dev`) and `cmake` to be installed. The `make check` command compiles and executes the tests. ```bash mkdir build cd build cmake .. make check -j 4 ``` -------------------------------- ### Create pybind11 Bindings for a C++ Function Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet demonstrates how to create Python bindings for the `add` function using `PYBIND11_MODULE`. It defines a module named 'example' and exposes the `add` function with an optional docstring, automatically inferring parameter and return types. ```cpp #include int add(int i, int j) { return i + j; } PYBIND11_MODULE(example, m) { m.doc() = "pybind11 example plugin"; // optional module docstring m.def("add", &add, "A function which adds two numbers"); } ``` -------------------------------- ### Initialize and Use FC Storage with Aliyun OSS (Java) Source: https://github.com/tianmiantech/wefe/blob/main/common/java/common-data-storage/README.md This example illustrates the initialization of the `FcStorage` instance with an `AliyunOssConfig`. It covers both the initial setup and re-initialization, followed by a bulk data insertion operation using `putAll()` for function compute scenarios. ```java // Initialize storage instance FcStorage.init(new AliyunOssConfig(...)); // Re-initialize storage with new configuration FcStorage.init(new AliyunOssConfig(...)); // Use storage FcStorage.getInstance().putAll(""); ``` -------------------------------- ### Install Docker Automatically on Ubuntu Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This snippet provides a shell command to automatically install Docker on Ubuntu using the official script with the Aliyun mirror for faster downloads. ```shell curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun ``` -------------------------------- ### Interactive Startup of Flow Project Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/flow.md This script outlines the steps to interactively start the 'flow' project. It covers activating the Python virtual environment, declaring necessary environment variables (PYTHONPATH, SPARK_HOME, JAVA_HOME, PATH), installing Python dependencies using `pip`, and launching the main application (`app_launcher.py`) in the background with logging enabled. ```shell # 激活虚拟环境 source [Virtual Env Dir]/bin/activate # 声明环境变量 export PYTHONPATH=[Root Dir] export SPARK_HOME=[Spark Dir] export JAVA_HOME=[JDK Dir] export PATH=$SPARK_HOME/bin:$JAVA_HOME/bin:$PATH # 下载环境依赖 pip install -r ${PYTHON_ROOT}/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple # 启动项目 cd [Root Dir]/flow nohup python3 app_launcher.py >> "[Log Dir]/console.log" 2>>"[Log Dir]/error.log" & ``` -------------------------------- ### Install Aliyun STS Python SDK Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/函数计算开发文档.md This command installs the `aliyun-python-sdk-sts` package using pip. This SDK is required for interacting with Alibaba Cloud's Security Token Service (STS) to obtain temporary access credentials. ```shell script pip install aliyun-python-sdk-sts ``` -------------------------------- ### Run fusion-service JAR Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/fusion.md This command starts the fusion-service application from its executable JAR file. Ensure the database is initialized and JDK 1.8 is installed. ```shell java -jar fusion-service.jar ``` -------------------------------- ### Importing and Executing a Pybind11 Module in Python Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This snippet demonstrates how to import a compiled C++ module (e.g., 'example') into a Python interactive session and call a simple function ('add') defined within it. It assumes the compiled module is accessible in the current directory. ```pycon $ python Python 2.7.10 (default, Aug 22 2015, 20:33:39) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import example >>> example.add(1, 2) 3L ``` -------------------------------- ### Run Wefe VisualFL Example Job Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This snippet demonstrates how to execute an example job on a deployed Wefe VisualFL machine. It activates the virtual environment, sets the `PYTHONPATH` to include VisualFL, and then runs a PaddlePaddle classification example script, specifying the master service address. ```bash cd {base_dir} source venv/bin/activate export PYTHONPATH=$PYTHONPATH:{base_dir}/VisualFL sh VisualFL/examples/paddle_clas/run.sh 127.0.0.1:10002 ``` -------------------------------- ### Interactive Startup Commands for Flow Project Source: https://github.com/tianmiantech/wefe/blob/main/flow/README.md This section provides a sequence of shell commands for manually starting the 'flow' project. It covers activating the Python virtual environment, setting critical environment variables (PYTHONPATH, SPARK_HOME, JAVA_HOME), installing Python dependencies, and finally launching the main application in the background. ```shell # 激活虚拟环境 source [Virtual Env Dir]/bin/activate # 声明环境变量 export PYTHONPATH=[Root Dir] export SPARK_HOME=[Spark Dir] export JAVA_HOME=[JDK Dir] export PATH=$SPARK_HOME/bin:$JAVA_HOME/bin:$PATH # 下载环境依赖 pip install -r ${PYTHON_ROOT}/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple # 启动项目 cd [Root Dir]/flow nohup python3 app_launcher.py >> "[Log Dir]/console.log" 2>>"[Log Dir]/error.log" & ``` -------------------------------- ### Install Docker on macOS using Homebrew Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This command installs Docker Desktop on macOS using Homebrew Cask, placing the application in the /Applications directory. ```shell brew install --cask --appdir=/Applications docker ``` -------------------------------- ### Manually Install Docker and Docker Compose on CentOS Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md Detailed steps to manually install Docker CE, Docker CLI, containerd.io, and Docker Compose on CentOS. This includes installing necessary dependencies, adding the Docker CE repository, installing packages, and verifying the installation. ```shell # ---------- 安装 docker ---------- yum install -y yum-utilsdevice-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.13-3.1.el7.x86_64.rpm yum install docker-ce docker-ce-cli containerd.io # 检测是否安装成功 docker -v # ---------- 安装 docker compose ---------- # 可能会出现连接超时的情况,请耐心的尝试几次 wget https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-x86_64 mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose # 添加执行权限 chmod +x /usr/local/bin/docker-compose # 检测是否安装成功 docker-compose -version ``` -------------------------------- ### Start Wefe VisualFL Services (Standalone) Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This command starts all services for a standalone Wefe VisualFL deployment, referencing the configuration in `standalone_template.yaml`. This provides a convenient way to manage services via the deploy toolkit, as an alternative to using scripts. ```bash wefe_visualfl_deploy services all start standalone_template.yaml ``` -------------------------------- ### Basic pybind11 Extension Setup in setup.py Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This Python snippet demonstrates a fundamental `setup.py` configuration for building C++ extensions using `pybind11` and `setuptools`. It utilizes `Pybind11Extension` to define an extension module, specifying source files for compilation. This setup is suitable for projects integrating C++ code with Python. ```python from glob import glob from setuptools import setup from pybind11.setup_helpers import Pybind11Extension ext_modules = [ Pybind11Extension( "python_example", sorted(glob("src/*.cpp")), # Sort source files for reproducibility ), ] setup( ..., ext_modules=ext_modules ) ``` -------------------------------- ### Install Docker Automatically on CentOS Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This snippet provides a shell command to automatically install Docker on CentOS using the official script with the Aliyun mirror for faster downloads. ```shell curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun ``` -------------------------------- ### Define a Simple C++ Function for pybind11 Binding Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet defines a basic `add` function that takes two integers and returns their sum. This function will be used as an example to demonstrate how to create Python bindings using pybind11. ```cpp int add(int i, int j) { return i + j; } ``` -------------------------------- ### rundis-cli install Subcommand Options Source: https://github.com/tianmiantech/wefe/blob/main/deploy/WeFe隐私计算服务部署说明文档.md Explains various options for the `install` command, which initializes third-party software. It warns that this command is dangerous as it overwrites existing containers and causes data loss. Options include installing all, member-specific, center-specific, or specified services. ```bash rundis-cli install --all ``` ```bash rundis-cli install --member ``` ```bash rundis-cli install --center ``` ```bash rundis-cli install mysql clickhouse nacos redis mongo ``` -------------------------------- ### Compile pybind11 Test Cases on Windows Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This snippet outlines the steps to compile and run pybind11 test cases on Windows using Visual Studio 2015 or newer. It creates a Visual Studio project, compiles, and runs the target from the command line. Ensure Python binary and test cases match processor type and bitness. ```batch mkdir build cd build cmake .. cmake --build . --config Release --target check ``` -------------------------------- ### CMake Project with find_package(pybind11) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst Example CMakeLists.txt demonstrating how to use `find_package(pybind11)` to detect an external pybind11 installation and add a module using `pybind11_add_module`. ```cmake cmake_minimum_required(VERSION 3.4...3.18) project(example LANGUAGES CXX) find_package(pybind11 REQUIRED) pybind11_add_module(example example.cpp) ``` -------------------------------- ### Install Python Dependencies for Custom Function Compute Runtime Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/函数计算开发文档.md This snippet provides the `fun-install` commands to install Python dependencies for a custom function compute runtime. It uses Tsinghua University's PyPI mirror for faster downloads and includes a broader set of packages compared to the default runtime. ```shell script RUNTIME custom RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install six==1.15.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install cachetools==4.1.1 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install numba==0.50.1 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install numpy==1.19.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install pycryptodome==3.3.1 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install pycryptodomex==3.9.8 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install Deprecated==1.2.10 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install python-jose-cryptodome==1.3.2 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install scipy==1.5.1 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install psutil==5.7.2 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install peewee==3.13.3 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install requests==2.24.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install protobuf==3.12.2 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install PyMySQL==0.10.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install sklearn RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install grpcio==1.29.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install grpcio-tools==1.29.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install clickhouse-driver==0.1.4 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install tornado RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install tablestore==5.1.0 RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install aliyun-python-sdk-core RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install flask RUN fun-install pip -i https://pypi.tuna.tsinghua.edu.cn/simple install gunicorn ``` -------------------------------- ### Install pybind11 using CMake Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst Bash commands to build and install pybind11 using both classic CMake workflow and CMake 3.15+ commands. ```bash # Classic CMake cd pybind11 mkdir build cd build cmake .. make install ``` ```bash # CMake 3.15+ cd pybind11 cmake -S . -B build cmake --build build -j 2 # Build on 2 cores cmake --install build ``` -------------------------------- ### Install Python Dependencies for Function Compute (Python 3 Runtime) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/函数计算开发文档.md This snippet lists the Python packages required for the function compute environment when using the default Python 3 runtime. These dependencies are installed using `fun-install pip install` commands. ```shell script RUNTIME python3 RUN fun-install pip install six==1.15.0 RUN fun-install pip install cachetools==4.1.1 RUN fun-install pip install numba==0.50.1 RUN fun-install pip install pycryptodome==3.3.1 RUN fun-install pip install pycryptodomex==3.9.8 RUN fun-install pip install Deprecated==1.2.10 RUN fun-install pip install python-jose-cryptodome==1.3.2 RUN fun-install pip install psutil==5.7.2 RUN fun-install pip install cloudpickle RUN fun-install pip install peewee==3.13.3 RUN fun-install pip install requests==2.24.0 RUN fun-install pip install protobuf==3.12.2 RUN fun-install pip install PyMySQL==0.10.0 ``` -------------------------------- ### Install pybind11 via Python Package Index (PyPI) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/installing.rst Install pybind11 as a Python package using pip. This method provides pybind11 in a standard Python package format. It includes options for a standard installation or a global installation, with a caution against global installation for system Python. ```bash pip install pybind11 ``` ```bash pip install "pybind11[global]" ``` -------------------------------- ### Start Wefe VisualFL Services (Cluster) Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This command starts all services for a Wefe VisualFL cluster deployment, referencing the configuration in `template.yaml`. This provides a convenient way to manage services across the cluster via the deploy toolkit, simplifying operations. ```bash wefe_visualfl_deploy services all start template.yaml ``` -------------------------------- ### Configure Optional Spark Settings for WeFe (wefe.cfg) Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This INI snippet provides optional configuration items in `wefe.cfg` to optimize Spark performance, recommended for servers with 32GB or more memory. These settings can improve modeling efficiency for larger datasets. ```ini # 若服务器内存 ≥ 32G,可酌情优化为如下配置,否则保持默认配置即可 SPARK_DRIVER_MEMORY=15g SPARK_DRIVER_MAXRESULTSIZE=2g SAPRK_NUM_EXECUTORS=6 SPARK_EXECUTOR_MEMORY=2g SPARK_EXECUTOR_CORES=1 SPARK_NUM_SLICES=32 ``` -------------------------------- ### Build Wefe VisualFL Deployment Package Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This snippet shows how to build the `visualfl_deploy` package from source. It navigates to the deployment tools directory, runs a build script, and then creates a source distribution (`.tar.gz`) ready for upload and installation. ```bash cd VisualFL/deploy_tools/visualfl_deploy python visualfl_deploy/_build.py cd .. pyhon setup.py sdist ``` -------------------------------- ### Python Block and Line Comment Examples Source: https://github.com/tianmiantech/wefe/blob/main/documents/code_style/python_code_style.md Shows examples of multi-line block comments for complex operations and single-line comments for non-obvious code. This emphasizes the importance of explaining tricky parts of the code for better readability and maintainability. ```Python # We use a weighted dictionary search to find out where i is in # the array. We extrapolate position based on the largest num # in the array and the array size and then do binary search to # get the exact number. if i & (i-1) == 0: # true iff i is a power of 2 pass ``` -------------------------------- ### Inspecting Pybind11 Function Signatures with Keyword Arguments in Python Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This Python interactive session output shows how the 'help()' function reveals the named arguments ('i' and 'j') in the signature of a pybind11-bound C++ function, reflecting the metadata provided during binding. ```pycon >>> help(example) .... FUNCTIONS add(...) Signature : (i: int, j: int) -> int A function which adds two numbers ``` -------------------------------- ### Start Spring Boot Application Source: https://github.com/tianmiantech/wefe/blob/main/union/blockchain-data-sync/README.md Command to start the compiled blockchain-data-sync Spring Boot application. This command executes the JAR file, launching the service. ```bash java -jar blockchain-data-sync.jar ``` -------------------------------- ### Compile Project with build.sh Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/go/src/README.md Executes the build.sh script to compile the go-utils dynamic link library. This process requires a Go environment to be installed on the system. ```shell # 编译go-utils动态链接库 sh build.sh ``` -------------------------------- ### Install Wefe VisualFL Deploy Toolkit Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This command sequence demonstrates how to install the `visualfl_deploy` toolkit within a Python virtual environment. It creates a new virtual environment, activates it, upgrades pip, and then installs the `visualfl_deploy` package from a local tar.gz file. ```bash # ceate a python virtual envirement (recommanded) or use an exist one. cd {base_dir} python -m venv venv source venv/bin/activate python -m pip install -U pip && python -m pip install visualfl_deploy-1.0.tar.gz ``` -------------------------------- ### Inspecting Pybind11 Function Signatures with Default Arguments in Python Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This Python interactive session output illustrates how the 'help()' function displays the default argument values in the signature of a pybind11-bound C++ function, making them visible to Python users. ```pycon >>> help(example) .... FUNCTIONS add(...) Signature : (i: int = 1, j: int = 2) -> int A function which adds two numbers ``` -------------------------------- ### Configure Required WeFe Network Settings (wefe.cfg) Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This INI snippet shows the mandatory configuration items in `wefe.cfg` for setting the local intranet and extranet IP addresses, which are crucial for WeFe's communication. ```ini # 本机内网 IP INTRANET_IP=0.0.0.0 # 本机外网 IP EXTRANET_IP=0.0.0.0 ``` -------------------------------- ### Python Class Docstring Example Source: https://github.com/tianmiantech/wefe/blob/main/documents/code_style/python_code_style.md Illustrates a docstring for a Python class, including a summary, longer description, and an 'Attributes' section for public properties. It also shows the `__init__` method and a public method with their respective docstrings, following standard documentation practices. ```Python class SampleClass(object): """Summary of class here. Longer class information.... Longer class information.... Attributes: likes_spam: A boolean indicating if we like SPAM or not. eggs: An integer count of the eggs we have laid. """ def __init__(self, likes_spam=False): """Inits SampleClass with blah.""" self.likes_spam = likes_spam self.eggs = 0 def public_method(self): """Performs operation blah.""" ``` -------------------------------- ### Standard pybind11 Header and Namespace Inclusion Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet shows the standard header inclusion and namespace alias for pybind11. It's a common prerequisite for all pybind11 binding code, ensuring access to the library's functionalities. ```cpp #include namespace py = pybind11; ``` -------------------------------- ### Start All Member Node Services with rundis-cli Source: https://github.com/tianmiantech/wefe/blob/main/deploy/WeFe隐私计算服务部署说明文档.md Starts all services relevant to a member node using the `rundis-cli` tool. ```bash rundis-cli upstart --member ``` -------------------------------- ### Install pybind11 via vcpkg Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/installing.rst Install pybind11 using Microsoft's vcpkg dependency manager. This involves cloning the vcpkg repository, bootstrapping it, integrating it, and then installing pybind11 through vcpkg. This method is maintained by Microsoft and community contributors. ```bash git clone https://github.com/Microsoft/vcpkg.git cd vcpkg ./bootstrap-vcpkg.sh ./vcpkg integrate install vcpkg install pybind11 ``` -------------------------------- ### Calling Pybind11 Functions with Keyword Arguments in Python Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst After a C++ function is bound with named arguments using 'py::arg', this Python snippet demonstrates how to call that function using keyword arguments, making the function calls more explicit and readable. ```pycon >>> import example >>> example.add(i=1, j=2) 3L ``` -------------------------------- ### CMake Project with FindPython and pybind11 Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst CMakeLists.txt example showing how to integrate pybind11 when using CMake's `FindPython` module (3.12+), which provides improved Python discovery and targets. ```cmake cmake_minimum_required(VERSION 3.15...3.19) project(example LANGUAGES CXX) find_package(Python COMPONENTS Interpreter Development REQUIRED) find_package(pybind11 CONFIG REQUIRED) # or add_subdirectory(pybind11) pybind11_add_module(example example.cpp) ``` -------------------------------- ### Install pybind11 via Homebrew or Linuxbrew Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/installing.rst Install pybind11 using the brew package manager, which includes Homebrew for macOS and Linuxbrew for Linux. This provides a simple command-line installation for users of these package managers. ```bash brew install pybind11 ``` -------------------------------- ### rundis-cli upstart Subcommand Options Source: https://github.com/tianmiantech/wefe/blob/main/deploy/WeFe隐私计算服务部署说明文档.md Explains various options for the `upstart` command, allowing users to start all services, member node services, federal center services, or specific services. ```bash rundis-cli upstart --all ``` ```bash rundis-cli upstart --member ``` ```bash rundis-cli upstart --center ``` ```bash rundis-cli upstart mysql clickhouse nacos ``` -------------------------------- ### Initialize and Use Persistent Storage with ClickHouse (Java) Source: https://github.com/tianmiantech/wefe/blob/main/common/java/common-data-storage/README.md This snippet demonstrates how to initialize the `PersistentStorage` instance using a `ClickhouseConfig` and then store a `DataItemModel`. It shows both initial setup and re-initialization with new configurations, followed by a data insertion operation. ```java // Initialize storage instance PersistentStorage.init(new ClickhouseConfig("127.0.0.1",8123,"user","pasdword")); // Re-initialize storage with new configuration PersistentStorage.init(new ClickhouseConfig("127.0.0.1",8123,"user","pasdword")); // Use storage PersistentStorage.getInstance().put("wefe","test",new DataItemModel("a","123")); ``` -------------------------------- ### Python Function Docstring Example (NumPy Style) Source: https://github.com/tianmiantech/wefe/blob/main/documents/code_style/python_code_style.md Demonstrates a NumPy-style docstring for a Python function, detailing its purpose, parameters, return values, and potential exceptions. This format is recommended for clear and consistent function documentation, aiding automatic documentation generation. ```Python def fetch_bigtable_rows(big_table, keys, other_silly_variable=None): """ Fetches rows from a Bigtable. Retrieves rows pertaining to the given keys from the Table instance represented by big_table. Silly things may happen if other_silly_variable is not None. Parameters ---------- keys: string xxx other_silly_variable: string xxx Returns ------- A dict mapping keys to the corresponding table row data fetched. Each row is represented as a tuple of strings. For example: {'Serak': ('Rigel VII', 'Preparer'), 'Zim': ('Irk', 'Invader'), 'Lrrr': ('Omicron Persei 8', 'Emperor')} If a key from the keys argument is missing from the dictionary, then that row was not found in the table. Raises: ---------- IOError: An error occurred accessing the bigtable.Table object. """ pass ``` -------------------------------- ### Configure and Install GMP Library Locally Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/GPU/CGBN/docs/README.md Instructions for configuring and installing the GNU Multiple Precision Library (GMP) locally, which is a prerequisite for CGBN. This command configures GMP to be installed in the directory specified by the GMP_HOME environment variable. ```Shell ./configure --prefix=$GMP_HOME ``` -------------------------------- ### Install pybind11 as a Git Submodule Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/installing.rst This method allows integrating pybind11 directly into a Git project as a submodule. It's suitable for projects managing dependencies within their repository. The commands add the pybind11 repository and initialize it, with options for relative or full URLs. ```bash git submodule add -b stable ../../pybind/pybind11 extern/pybind11 git submodule update --init ``` -------------------------------- ### Run fusion-website in development mode Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/fusion.md This command starts the fusion-website in development mode, typically with hot-reloading. Requires Node.js and npm. ```shell npm run dev ``` -------------------------------- ### Build fusion-service with Maven Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/fusion.md This command compiles and packages the fusion-service module using Maven, skipping tests. Ensure Maven is installed and configured on the server. ```shell mvn clean install -Dmaven.test.skip=true -am -pl fusion/fusion-service ``` -------------------------------- ### Execute Wefe VisualFL Cluster Deployment Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This command initiates the deployment process for a Wefe VisualFL cluster. It uses the configuration specified in `template.yaml`. Ensure the template has been reviewed and modified for your specific cluster setup before execution. ```bash wefe_visualfl_deploy deploy deploy --config template.yaml ``` -------------------------------- ### Compile Go Utility Package Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/函数计算开发文档.md These shell commands demonstrate how to compile the Go utility package. It navigates to the source directory, downloads Go module dependencies if needed, and then executes the `build.sh` script to compile the dynamic link library. ```shell cd business/go/src # 首次编译需要调用go mod download,自动下载go.mod的依赖包 go mod download # 编译go-utils动态链接库 sh build.sh ``` -------------------------------- ### Manage Fisco Services with Docker Source: https://github.com/tianmiantech/wefe/blob/main/deploy/WeFe隐私计算服务部署说明文档.md Fisco services are an exception and must be managed using native Docker commands for stopping, starting, and restarting containers, unlike other services managed by `rundis-cli`. ```bash # 停止 docker stop fisco-console fisco-node1 fisco-node0 # 启动 docker start fisco-console fisco-node1 fisco-node0 # 重启 docker restart fisco-console fisco-node1 fisco-node0 ``` -------------------------------- ### Generate Wefe VisualFL Deployment Template (Standalone) Source: https://github.com/tianmiantech/wefe/blob/main/VisualFL/deploy_tools/README.md This command generates a configuration template for a standalone deployment of Wefe VisualFL. The output file `standalone_template.yaml` can then be modified to customize the deployment settings for a single machine setup. ```bash wefe_visualfl_deploy template standalone ``` -------------------------------- ### C++ Function Definition with Default Arguments Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ code defines a simple 'add' function with default values for its integer parameters 'i' and 'j'. This function will be subsequently bound to Python using pybind11. ```cpp int add(int i = 1, int j = 2) { return i + j; } ``` -------------------------------- ### Manual pybind11 Package Build and Upload Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/release.rst Outlines the steps for manually building pybind11 source distributions and wheels, and subsequently uploading them to PyPI using `twine`. This method is generally not recommended due to potential 'dirty' local environments. ```Bash python3 -m pip install build python3 -m build PYBIND11_SDIST_GLOBAL=1 python3 -m build twine upload dist/* ``` -------------------------------- ### Accessing C++ Exported Variables in Python Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This Python interactive session shows how to access variables that have been exported from a C++ module via pybind11. The exported C++ variables appear as attributes of the imported Python module. ```pycon >>> import example >>> example.the_answer 42 >>> example.what 'World' ``` -------------------------------- ### Pybind11 CMake Helper Functions Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst Documentation for pybind11's CMake helper functions, `pybind11_strip` for stripping targets and `pybind11_extension` for setting correct module extensions. ```APIDOC pybind11_strip(target): Strips a target (uses CMAKE_STRIP after the target is built) pybind11_extension(target): Sets the correct extension (with SOABI) for a target. ``` -------------------------------- ### Start PaddleServing Inference Service (GPU/CPU) Source: https://github.com/tianmiantech/wefe/blob/main/serving/serving-service/Wefe深度学习Serving部署.md These commands initiate the PaddleServing inference service. Navigate to the exported model directory ('inference_model/yolov3_mobilenet_v1_roadsign/') before running. Options are provided for both GPU-accelerated and CPU-only deployment, specifying the model and port. The 'sudo' prefix might be required depending on permissions. ```python python -m paddle_serving_server_gpu.serve --model serving_server --port 9393 --gpu_ids 0 ``` ```python python3 -m paddle_serving_server.serve --model serving_server --port 9393 ``` -------------------------------- ### Configure Docker Daemon Persistence Directory on CentOS Source: https://github.com/tianmiantech/wefe/blob/main/docs/install/install.md This JSON snippet shows how to configure the Docker daemon to change its persistence directory to a larger mounted disk, recommending a size greater than 100GB, by modifying or creating `/etc/docker/daemon.json`. ```json { "graph": "/data/wefe/docker-compose", "live-restore": true } ``` -------------------------------- ### Java Example: Initiating Federated Prediction with WeFe Serving SDK Source: https://github.com/tianmiantech/wefe/blob/main/serving/README.md This Java code demonstrates the end-to-end process of making federated prediction calls using the WeFe Serving SDK. It includes static initialization of the SDK with member ID and RSA keys, setting up a list of provider parameters, preparing feature data for prediction, and then executing predictions separately for a 'promoter' role and a 'provider' role, showcasing the different prediction flows. ```Java import com.welab.wefe.serving.sdk.dto.PredictResult; import com.welab.wefe.serving.sdk.dto.ProviderParams; import org.apache.commons.compress.utils.Lists; import java.util.HashMap; import java.util.List; import java.util.Map; /** * 该类主要演示如和发起调用推理预测 * */ public class Example { static { try { Launcher.init("memberId", "rsaPrivateKey", "rsaPublicKey"); } catch (Exception e) { e.printStackTrace(); } } public static void main(String[] args) { List providers = Lists.newArrayList(); providers.add(ProviderParams.of("member01", "https://10.0.0.1/provider")); providers.add(ProviderParams.of("member02", "https://10.0.0.2/provider")); Map featureData = new HashMap<>(16); featureData.put("x0", 0.100016); featureData.put("x1", 1.210); featureData.put("x2", 2.321); featureData.put("x3", 3.432); featureData.put("x4", 4.543); featureData.put("x5", 5.654); PredictParams predictParams = PredictParams.of("15555555555", featureData); try { /** * promoter */ ExamplePromoterPredicter promoter = new ExamplePromoterPredicter("modelId", predictParams, new JSONObject(), providers, "memberId"); PredictResult promoterResult = promoter.predict(); System.err.println(JSON.toJSONString(promoterResult)); /** * provider */ ExampleProviderPredicter provider = new ExampleProviderPredicter( FederatedParams.of("", "modelId-02", "memberId"), predictParams, new JSONObject()); PredictResult providerResult = provider.predict(); System.err.println(JSON.toJSONString(providerResult)); } catch (Exception e) { e.printStackTrace(); } System.err.println("over"); } } ``` -------------------------------- ### Build Serving Service Backend with Maven Source: https://github.com/tianmiantech/wefe/blob/main/serving/README.md This command builds the `serving-service` module, which is the backend of the Serving platform, using Maven. The `-Dmaven.test.skip=true` flag ensures that tests are skipped during the build process, and `-am -pl serving/serving-service` specifies to build the `serving-service` project and its required modules. ```Maven mvn clean install -Dmaven.test.skip=true -am -pl serving/serving-service ``` -------------------------------- ### Build fusion-website for production Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/fusion.md This command compiles and bundles the fusion-website for deployment, specifically for the 'dev' environment. Requires Node.js and npm. ```shell npm run build -- dev ``` -------------------------------- ### Shorthand Notation for Pybind11 Default Arguments (C++) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ code provides a more compact syntax for specifying default arguments when binding functions with pybind11, using the '_a=value' literal. This is an alternative to the 'py::arg("name") = value' syntax. ```cpp // regular notation m.def("add1", &add, py::arg("i") = 1, py::arg("j") = 2); // shorthand m.def("add2", &add, "i"_a=1, "j"_a=2); ``` -------------------------------- ### Compile and Package Java Project with Maven Source: https://github.com/tianmiantech/wefe/blob/main/union/blockchain-data-sync/README.md Instructions to compile and package the blockchain-data-sync project using Maven. This command should be executed in the project's root directory and requires a Maven environment on the deployment machine. ```bash mvn clean install -Dmaven.test.skip=true -am -pl union/blockchain-data-sync ``` -------------------------------- ### Shorthand Notation for Pybind11 Keyword Arguments (C++) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet introduces a more concise way to specify keyword arguments in pybind11 using the '_a' literal suffix. It requires including 'using namespace pybind11::literals;' to make the literal operator visible. ```cpp // regular notation m.def("add1", &add, py::arg("i"), py::arg("j")); // shorthand using namespace pybind11::literals; m.def("add2", &add, "i"_a, "j"_a); ``` -------------------------------- ### Set Go Environment Variables Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/fc/函数计算开发文档.md This snippet provides shell commands to set up essential Go environment variables. It defines `GOPATH` for Go workspaces, adds `GOPATH/bin` to the system `PATH`, enables Go modules, and configures `GOPROXY` for module downloads. ```shell export GOPATH=/Users/{xxx_your_go_path}/go export PATH=$PATH:$GOPATH/bin export GO111MODULE=on export GOPROXY=https://goproxy.io ``` -------------------------------- ### Example Debug Warning for Deprecated Placement-New __init__ Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/upgrade.rst This snippet shows a sample runtime warning that pybind11 emits in debug mode when an old-style placement-new __init__ method is detected in a bound class. This warning encourages migration to the new py::init() API. ```text pybind11-bound class 'mymodule.Foo' is using an old-style placement-new '__init__' which has been deprecated. See the upgrade guide in pybind11's docs. ``` -------------------------------- ### Defining Pybind11 Functions with Keyword Arguments (C++) Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ code shows how to bind a C++ function to Python using pybind11, explicitly naming its arguments ('i' and 'j') using 'py::arg'. This allows the function to be called with keyword arguments from Python, improving readability. ```cpp m.def("add", &add, "A function which adds two numbers", py::arg("i"), py::arg("j")); ``` -------------------------------- ### Exporting Variables from C++ to Python using Pybind11 Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet demonstrates how to expose C++ variables and objects as attributes within a Python module using pybind11's 'm.attr()' function. It shows assigning a literal integer and a 'py::object' created from a C++ string. ```cpp PYBIND11_MODULE(example, m) { m.attr("the_answer") = 42; py::object world = py::cast("World"); m.attr("what") = world; } ``` -------------------------------- ### Setting JAVA_HOME for Flow Script Startup Source: https://github.com/tianmiantech/wefe/blob/main/docs/system_framework/flow.md This snippet demonstrates how to declare the `JAVA_HOME` environment variable and append Java's binary directory to the system's `PATH`. This configuration is crucial for the script-based startup of the 'flow' project, ensuring Java executables are discoverable. ```shell # 声明 JAVA_HOME export JAVA_HOME=[JDK Dir] export PATH=$JAVA_HOME/bin:$PATH ``` -------------------------------- ### Binding C++ Functions with Default Arguments in Pybind11 Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/basics.rst This C++ snippet demonstrates how to explicitly specify default argument values when binding a C++ function to Python using pybind11. Since pybind11 cannot automatically extract these, they must be provided using 'py::arg("name") = value'. ```cpp m.def("add", &add, "A function which adds two numbers", py::arg("i") = 1, py::arg("j") = 2); ``` -------------------------------- ### CMake: Embedding Python Interpreter in C++ Executable Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This CMake example illustrates how to embed the Python interpreter within a C++ executable or library. By linking against the `pybind11::embed` target, all necessary Python headers and libraries are automatically attached, simplifying the setup for integrating Python into C++ applications. ```cmake cmake_minimum_required(VERSION 3.4...3.18) project(example LANGUAGES CXX) find_package(pybind11 REQUIRED) # or add_subdirectory(pybind11) add_executable(example main.cpp) target_link_libraries(example PRIVATE pybind11::embed) ``` -------------------------------- ### Handle pybind11 Extension Import for Classic setup_requires Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This Python snippet provides a fallback mechanism for `setup.py` when using the classic `setup_requires` approach. It attempts to import `Pybind11Extension` from `pybind11.setup_helpers` and falls back to `setuptools.Extension` if `pybind11` is not yet installed during the first pass of a two-phase build. ```python try: from pybind11.setup_helpers import Pybind11Extension except ImportError: from setuptools import Extension as Pybind11Extension ``` -------------------------------- ### Declaring Buffer Protocol Support with `py::class_` in pybind11 C++ Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/upgrade.rst This C++ example demonstrates a breaking change in pybind11 v2.0, where types providing buffer protocol access must now explicitly include `py::buffer_protocol()` as an argument to the `py::class_` constructor. This ensures future-proofing and improved efficiency for type definitions. ```cpp py::class_("Matrix", py::buffer_protocol()) .def(py::init<...>()) .def_buffer(...); ``` -------------------------------- ### Pybind11 Advanced CMake Interface Library Targets Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst Documentation for pybind11's modern CMake interface targets, providing granular control over compilation and linking requirements for extension modules and embedding. ```APIDOC pybind11::headers: Just the pybind11 headers and minimum compile requirements pybind11::python2_no_register: Quiets the warning/error when mixing C++14 or higher and Python 2 pybind11::pybind11: Python headers + pybind11::headers + pybind11::python2_no_register (Python 2 only) pybind11::python_link_helper: Just the "linking" part of pybind11:module pybind11::module: Everything for extension modules - pybind11::pybind11 + Python::Module (FindPython CMake 3.15+) or pybind11::python_link_helper pybind11::embed: Everything for embedding the Python interpreter - pybind11::pybind11 + Python::Embed (FindPython) or Python libs pybind11::lto / pybind11::thin_lto: An alternative to INTERPROCEDURAL_OPTIMIZATION for adding link-time optimization. pybind11::windows_extras: /bigobj and /mp for MSVC. pybind11::opt_size: /Os for MSVC, -Os for other compilers. Does nothing for debug builds. ``` -------------------------------- ### Configure pyproject.toml for PEP 518 Builds with pybind11 Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This TOML configuration defines the build system requirements for a Python project using PEP 518. It ensures that `setuptools`, `wheel`, and `pybind11` are available in a temporary virtual environment during the build process, allowing Pip to create a binary wheel. ```toml [build-system] requires = ["setuptools>=42", "wheel", "pybind11~=2.6.1"] build-backend = "setuptools.build_meta" ``` -------------------------------- ### Optimize Recompilation with naive_recompile for Faster Development Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This Python code illustrates how to use `SmartCompile` with `naive_recompile` from `pybind11.setup_helpers` to skip rebuilding C++ object files that are newer than their source. This optimization is particularly useful for rapid development cycles with `pip install -e .`, reducing build times by avoiding unnecessary recompilation of unchanged files. ```python from pybind11.setup_helpers import ParallelCompile, naive_recompile SmartCompile("NPY_NUM_BUILD_JOBS", needs_recompile=naive_recompile).install() ``` -------------------------------- ### Handling Wide Character Strings between Python and C++ Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/advanced/cast/strings.rst This snippet demonstrates how to pass Python `str` to C++ functions expecting `std::wstring` and return `std::wstring` to Python. It shows examples of setting and getting window text using Windows API functions, highlighting the automatic UTF-16/UTF-32 encoding/decoding. ```C++ #define UNICODE #include m.def("set_window_text", [](HWND hwnd, std::wstring s) { // Call SetWindowText with null-terminated UTF-16 string ::SetWindowText(hwnd, s.c_str()); } ); m.def("get_window_text", [](HWND hwnd) { const int buffer_size = ::GetWindowTextLength(hwnd) + 1; auto buffer = std::make_unique< wchar_t[] >(buffer_size); ::GetWindowText(hwnd, buffer.data(), buffer_size); std::wstring text(buffer.get()); // wstring will be converted to Python str return text; } ); ``` -------------------------------- ### CMake: Building pybind11 Extension with `add_library` Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/compiling.rst This CMake example demonstrates how to build a pybind11 extension module using standard CMake commands like `add_library` and `target_link_libraries`, offering an alternative to `pybind11_add_module`. It shows how to link against `pybind11::module`, `pybind11::lto`, and `pybind11::windows_extras`, and set target properties for visibility. ```cmake cmake_minimum_required(VERSION 3.4) project(example LANGUAGES CXX) find_package(pybind11 REQUIRED) # or add_subdirectory(pybind11) add_library(example MODULE main.cpp) target_link_libraries(example PRIVATE pybind11::module pybind11::lto pybind11::windows_extras) pybind11_extension(example) pybind11_strip(example) set_target_properties(example PROPERTIES CXX_VISIBILITY_PRESET "hidden" CUDA_VISIBILITY_PRESET "hidden") ``` -------------------------------- ### Updated `py::metaclass` Usage for Static Properties in pybind11 C++ Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/upgrade.rst This C++ example illustrates the updated usage of `py::metaclass` in pybind11 v2.1. It shows that the zero-parameter `py::metaclass()` is deprecated as static properties now work by default. A new one-parameter `py::metaclass(python_type)` is introduced for advanced cases requiring custom metaclasses. ```cpp // old -- emits a deprecation warning py::class_(m, "Foo", py::metaclass()) .def_property_readonly_static("foo", ...); // new -- static properties work without the attribute py::class_(m, "Foo") .def_property_readonly_static("foo", ...); // new -- advanced feature, override pybind11's default metaclass py::class_(m, "Bar", py::metaclass(custom_python_type)) ... ``` -------------------------------- ### Enforcing Stricter `py::init` Constructor Bindings in C++ Source: https://github.com/tianmiantech/wefe/blob/main/common/python/calculation/acceleration/libs/tools/pybind11/docs/upgrade.rst This C++ example demonstrates the stricter compile-time checks for `py::init` constructors in pybind11. It shows that an exact match for lvalue references is required, preventing bindings that could lead to unexpected runtime behavior. A `const T &` can still bind to an rvalue using `py::init()`. ```cpp struct Example { Example(int &); }; py::class_(m, "Example") .def(py::init()); // OK, exact match // .def(py::init()); // compile-time error, mismatch ```