### Install and Configure Meta Service

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Installs the meta service on the meta node, copies its configuration files, and updates them with cluster and mgmtd connection details. It also demonstrates uploading the meta service configuration to mgmtd and starting the service.

```bash
cp ~/3fs/build/bin/meta_main /opt/3fs/bin
cp ~/3fs/configs/{meta_main_launcher.toml,meta_main.toml,meta_main_app.toml} /opt/3fs/etc
```

```toml
# meta_main_app.toml
node_id = 100

# meta_main_launcher.toml
cluster_id = "stage"

[mgmtd_client]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

# meta_main.toml
[server.mgmtd_client]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

[common.monitor.reporters.monitor_collector]
remote_ip = "192.168.1.1:10000"

[server.fdb]
clusterFile = '/opt/3fs/etc/fdb.cluster'
```

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type META --file /opt/3fs/etc/meta_main.toml"
```

```bash
cp ~/3fs/deploy/systemd/meta_main.service /usr/lib/systemd/system
systemctl start meta_main
```

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"
```

--------------------------------

### Install and Configure mgmtd Service

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Installs the mgmtd service on the meta node, copies its configuration files, and updates them with cluster information. It also includes the initialization of the cluster using admin_cli and starting the mgmtd service via systemd.

```bash
cp ~/3fs/build/bin/mgmtd_main /opt/3fs/bin
cp ~/3fs/configs/{mgmtd_main.toml,mgmtd_main_launcher.toml,mgmtd_main_app.toml} /opt/3fs/etc
```

```toml
# mgmtd_main_app.toml
node_id = 1

# mgmtd_main_launcher.toml
cluster_id = "stage"

[fdb]
clusterFile = '/opt/3fs/etc/fdb.cluster'

# mgmtd_main.toml
[common.monitor.reporters.monitor_collector]
remote_ip = "192.168.1.1:10000"
```

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16"
```

```bash
cp ~/3fs/deploy/systemd/mgmtd_main.service /usr/lib/systemd/system
systemctl start mgmtd_main
```

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"
```

--------------------------------

### Start FUSE Client Service

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Copies the systemd service file for the FUSE client and starts the service.

```bash
cp ~/3fs/deploy/systemd/hf3fs_fuse_main.service /usr/lib/systemd/system
systemctl start hf3fs_fuse_main
```

--------------------------------

### Start Storage Service

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Copies the systemd service file for the storage service to the systemd directory and then starts the service. This command assumes the service file has been correctly placed.

```bash
rsync -avz meta:~/3fs/deploy/systemd/storage_main.service /usr/lib/systemd/system
systemctl start storage_main
```

--------------------------------

### admin_cli Commands

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Provides documentation for key admin_cli commands used in the 3fs installation process, including initializing a cluster and listing nodes.

```APIDOC
admin_cli:
  Usage: /opt/3fs/bin/admin_cli -cfg <config_file> [OPTIONS] <command> [args...]

  Commands:
    help
      Displays general help or help for a specific command.

    init-cluster --mgmtd <mgmtd_config_path> <chain_table_id> <chunk_size> <file_strip_size>
      Initializes a new 3fs cluster.
      Parameters:
        --mgmtd: Path to the mgmtd configuration file.
        chain_table_id: The ID for the chain table.
        chunk_size: The chunk size in bytes.
        file_strip_size: The file strip size.

    list-nodes
      Lists all nodes currently registered in the cluster.
      Options:
        --config.mgmtd_client.mgmtd_server_addresses <addresses>
          A list of mgmtd server addresses to connect to.

    set-config --type <config_type> --file <config_file>
      Uploads a configuration file to the mgmtd service.
      Parameters:
        --type: The type of configuration (e.g., META).
        --file: The path to the configuration file.
```

--------------------------------

### Storage Service Configuration Examples

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Provides example TOML configurations for the storage service, including cluster and management details, and target data paths. These files need to be updated with specific cluster information.

```toml
cluster_id = "stage"

[mgmtd_client]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

[server.mgmtd]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

[common.monitor.reporters.monitor_collector]
remote_ip = "192.168.1.1:10000"

[server.targets]
target_paths = ["/storage/data1/3fs","/storage/data2/3fs","/storage/data3/3fs","/storage/data4/3fs","/storage/data5/3fs","/storage/data6/3fs","/storage/data7/3fs","/storage/data8/3fs","/storage/data9/3fs","/storage/data10/3fs","/storage/data11/3fs","/storage/data12/3fs","/storage/data13/3fs","/storage/data14/3fs","/storage/data15/3fs","/storage/data16/3fs",]
```

--------------------------------

### Monitor Service Systemd Unit

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Installs and starts the monitor service using a systemd unit file. This ensures the monitor service runs automatically and can be managed by systemd.

```bash
mkdir -p /opt/3fs/{bin,etc}
mkdir -p /var/log/3fs
cp ~/3fs/build/bin/monitor_collector_main /opt/3fs/bin
cp ~/3fs/configs/monitor_collector_main.toml /opt/3fs/etc
cp ~/3fs/deploy/systemd/monitor_collector_main.service /usr/lib/systemd/system
systemctl start monitor_collector_main
```

--------------------------------

### Install P Framework

Source: https://github.com/deepseek-ai/3fs/blob/main/specs/README.md

Installs the P framework, a prerequisite for building and testing the 3fs project. This can be done via an official guide or by using the dotnet CLI if .NET is already installed.

```Shell
dotnet tool restore
```

--------------------------------

### List Chains and Chain Tables

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Verifies the successful upload of chains and chain tables by listing them using the admin CLI.

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-chains"
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-chain-tables"
```

--------------------------------

### Install and Configure admin_cli

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Installs the admin_cli tool on all nodes, copies configuration files, and updates the admin_cli.toml file with cluster details. This tool is essential for managing the 3fs cluster.

```bash
mkdir -p /opt/3fs/{bin,etc}
rsync -avz meta:~/3fs/build/bin/admin_cli /opt/3fs/bin
rsync -avz meta:~/3fs/configs/admin_cli.toml /opt/3fs/etc
rsync -avz meta:/etc/foundationdb/fdb.cluster /opt/3fs/etc
```

```toml
cluster_id = "stage"

[fdb]
clusterFile = '/opt/3fs/etc/fdb.cluster'
```

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml help
```

--------------------------------

### Upload Chains to Mgmtd

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Uploads the generated chain definitions to the mgmtd service using the admin CLI.

```bash
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") "upload-chains output/generated_chains.csv"
```

--------------------------------

### ClickHouse Table Creation

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Imports the SQL file into ClickHouse to create necessary tables for monitoring metrics. This is a prerequisite for the monitor service.

```bash
clickhouse-client -n < ~/3fs/deploy/sql/3fs-monitor.sql
```

--------------------------------

### Copy Storage Binaries and Configs

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Copies the storage service executable (`storage_main`) to the bin directory and essential configuration files to the etc directory using rsync.

```bash
rsync -avz meta:~/3fs/build/bin/storage_main /opt/3fs/bin
rsync -avz meta:~/3fs/configs/{storage_main_launcher.toml,storage_main.toml,storage_main_app.toml} /opt/3fs/etc
```

--------------------------------

### Format and Mount Disks

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Formats 16 NVMe SSDs as XFS, assigns labels, and mounts them to the specified directories. It also creates the necessary data and log directories for the 3fs storage service.

```bash
mkdir -p /storage/data{1..16}
mkdir -p /var/log/3fs
for i in {1..16};do mkfs.xfs -L data${i} -s size=4096 /dev/nvme${i}n1;mount -o noatime,nodiratime -L data${i} /storage/data${i};done
mkdir -p /storage/data{1..16}/3fs
```

--------------------------------

### Install Dependencies

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/data_placement/README.md

Installs the necessary Python packages for the data placement problem using a requirements file. This is a prerequisite for generating the chain table.

```bash
cd deploy/data_placement
pip install -r requirements.txt
```

--------------------------------

### Copy FUSE Client Binaries and Configs

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Copies the FUSE client executable and its configuration files to the appropriate directories.

```bash
cp ~/3fs/build/bin/hf3fs_fuse_main /opt/3fs/bin
cp ~/3fs/configs/{hf3fs_fuse_main_launcher.toml,hf3fs_fuse_main.toml,hf3fs_fuse_main_app.toml} /opt/3fs/etc
```

--------------------------------

### Create Storage Targets

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Executes the generated commands to create storage targets on the 3fs cluster. It uses the admin CLI with the previously saved admin token.

```bash
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") < output/create_target_cmd.txt
```

--------------------------------

### Create FUSE Mount Point

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Creates the directory that will serve as the mount point for the 3fs FUSE client.

```bash
mkdir -p /3fs/stage
```

--------------------------------

### Configure FUSE Client

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Configures the main FUSE client settings, including mgmtd server addresses and monitor reporter details.

```toml
[mgmtd]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

[common.monitor.reporters.monitor_collector]
remote_ip = "192.168.1.1:10000"
```

--------------------------------

### Upload Chain Table to Mgmtd

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Uploads the generated chain table to the mgmtd service, specifying a description for the stage.

```bash
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") "upload-chain-table --desc stage 1 output/generated_chain_table.csv"
```

--------------------------------

### Install Build Prerequisites for Ubuntu 22.04

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Installs essential development libraries and tools required for building the 3FS project on Ubuntu 22.04 using apt.

```bash
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev gcc-12 g++-12 libboost-all-dev build-essential
```

--------------------------------

### Verify Storage Node Status

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Executes the `list-nodes` command using `admin_cli` to check if the newly started storage service has successfully joined the cluster and is recognized by the management daemon.

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"
```

--------------------------------

### Install Build Prerequisites for openEuler 2403sp1

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Installs necessary development packages for building 3FS on openEuler 2403sp1 using yum.

```bash
yum install cmake libuv-devel lz4-devel xz-devel double-conversion-devel libdwarf-devel libunwind-devel \
    libaio-devel gflags-devel glog-devel gtest-devel gmock-devel clang-tools-extra clang lld \
    gperftools-devel gperftools openssl-devel gcc gcc-c++ boost-devel
```

--------------------------------

### Check FUSE Mount

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Verifies that the 3FS file system has been successfully mounted at the specified mount point.

```bash
mount | grep '/3fs/stage'
```

--------------------------------

### Troubleshoot Common Deployment Issues

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Offers steps for troubleshooting common deployment errors in 3fs, including checking log messages in `stdout/stderr` via `journalctl`, examining log files in `/var/log/3fs/`, and ensuring the log directory exists.

```bash
journalctl
```

--------------------------------

### Troubleshoot admin_cli init-cluster Error

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Provides guidance on resolving errors when initializing a cluster with `admin_cli init-cluster`, specifically addressing issues with `mgmtd_main.toml` and the need to clear data and re-initialize.

```bash
admin_cli init-cluster
```

--------------------------------

### Build Single-Node Cluster Requirements

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Explains the minimum requirements for building a single-node cluster in 3fs, highlighting the need for at least two storage services for data replication and how to bypass limitations in test environments.

```bash
--num-nodes=1
```

--------------------------------

### Clone and Initialize 3FS Repository

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Clones the 3FS repository from GitHub and initializes its submodules and applies patches. This is the initial setup step for working with the 3FS codebase.

```bash
git clone https://github.com/deepseek-ai/3fs
cd 3fs
git submodule update --init --recursive
./patches/apply.sh
```

--------------------------------

### Build hf3fs_cli

Source: https://github.com/deepseek-ai/3fs/blob/main/hf3fs_utils/README.md

Builds the hf3fs_cli tool as a Python wheel package using the setup script.

```bash
python3 setup_hf3fs_utils.py bdist_wheel
```

--------------------------------

### Create Admin User

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Adds a root admin user to the 3fs system. The admin token generated should be saved for subsequent operations.

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "user-add --root --admin 0 root"
```

--------------------------------

### Configure FUSE Client Launcher

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Sets essential parameters for the FUSE client launcher, including cluster ID, mount point, token file location, and mgmtd server addresses.

```toml
cluster_id = "stage"
mountpoint = '/3fs/stage'
token_file = '/opt/3fs/etc/token.txt'

[mgmtd_client]
mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]
```

--------------------------------

### Upload FUSE Client Config to Mgmtd

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Uploads the FUSE client's configuration file to the mgmtd service using the admin CLI.

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type FUSE --file /opt/3fs/etc/hf3fs_fuse_main.toml"
```

--------------------------------

### Upload Configuration to Mgmtd

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Uses the `admin_cli` tool to upload the storage service configuration file (`storage_main.toml`) to the management daemon (mgmtd). This ensures the cluster is aware of the new storage node's configuration.

```bash
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml"
```

--------------------------------

### Monitor Service Configuration

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Configures the monitor service to report metrics to a ClickHouse database. This involves specifying the database connection details in the `monitor_collector_main.toml` file.

```toml
[server.monitor_collector.reporter]
type = 'clickhouse'

[server.monitor_collector.reporter.clickhouse]
db = '3fs'
host = '<CH_HOST>'
passwd = '<CH_PASSWD>'
port = '<CH_PORT>'
user = '<CH_USER>'
```

--------------------------------

### Install Build Prerequisites for OpenCloudOS 9 and TencentOS 4

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Installs required development libraries and tools for building 3FS on OpenCloudOS 9 and TencentOS 4 using dnf.

```bash
dnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4 lz4-devel xz xz-devel \
    double-conversion-devel libdwarf-devel libunwind-devel libaio-devel gflags-devel glog-devel \
    libuv-devel gmock-devel gperftools gperftools-devel openssl-devel boost-static boost-devel mono-devel \
    libevent-devel libibverbs-devel numactl-devel python3-devel
```

--------------------------------

### Update Configuration Files

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Details the process for updating configuration files in 3fs, emphasizing that `mgmtd` manages all configuration and that modified files like `storage_main.toml` must be uploaded using `admin_cli set-config`.

```bash
admin_cli set-config
```

--------------------------------

### Generate Chain Table

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Generates chain table configuration files using a Python script. This process requires an incidence matrix generated in the previous step and defines parameters for chain and target IDs.

```bash
python ~/3fs/deploy/data_placement/src/setup/gen_chain_table.py \
   --chain_table_type CR --node_id_begin 10001 --node_id_end 10005 \
   --num_disks_per_node 16 --num_targets_per_disk 6 \
   --target_id_prefix 1 --chain_id_prefix 9 \
   --incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle
```

--------------------------------

### Generate Storage Target Commands

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Uses a Python script to generate commands for creating storage targets across multiple nodes. This involves specifying parameters like the number of nodes, replication factor, and targets per disk.

```bash
pip install -r ~/3fs/deploy/data_placement/requirements.txt
python ~/3fs/deploy/data_placement/src/model/data_placement.py \
   -ql -relax -type CR --num_nodes 5 --replication_factor 3 --min_targets_per_disk 6
```

--------------------------------

### Install 3FS Dependencies on Ubuntu

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Installs the necessary dependencies for building and running 3FS on Ubuntu 20.04. This includes build tools, libraries for networking, compression, and performance profiling.

```bash
# for Ubuntu 20.04.
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev libclang-rt-14-dev gcc-10 g++-10 libboost1.71-all-dev build-essential
```

--------------------------------

### Generate Chain Table and Commands

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/data_placement/README.md

Generates the chain table and commands for creating/removing storage targets. It takes the output from the data placement solution as input.

```bash
python src/setup/gen_chain_table.py --chain_table_type CR --node_id_begin 10001 --node_id_end 10005 --num_disks_per_node 16 --num_targets_per_disk 6 --incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle
```

--------------------------------

### Add Core Libraries

Source: https://github.com/deepseek-ai/3fs/blob/main/src/client/core/CMakeLists.txt

Adds the core-client and core-stub libraries to the project. This is a common build or setup command.

```bash
target_add_lib(core-client core-stub)
```

--------------------------------

### Increase AIO Max Requests

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/README.md

Adjusts the system's maximum number of asynchronous I/O requests to a higher value, which can improve performance for I/O-intensive applications like the 3fs storage service.

```bash
sysctl -w fs.aio-max-nr=67108864
```

--------------------------------

### 3fs CRAQ Chain Replication Example

Source: https://github.com/deepseek-ai/3fs/blob/main/docs/design_notes.md

Illustrates a typical chain replication setup for data placement in 3fs using CRAQ. It shows how storage targets are organized into chains for replicating file chunks across multiple nodes and SSDs.

```APIDOC
Chain Table Structure:

| Chain | Version | Target 1 (head) | Target 2 | Target 3 (tail) |
| :---: | :-----: | :-------------: | :------: | :-------------: |
|   1   |    1    |      `A1`       |   `B1`   |      `C1`       |
|   2   |    1    |      `D1`       |   `E1`   |      `F1`       |
|   3   |    1    |      `A2`       |   `B2`   |      `C2`       |
|   4   |    1    |      `D2`       |   `E2`   |      `F2`       |
|   5   |    1    |      `A3`       |   `B3`   |      `C3`       |
|   6   |    1    |      `D3`       |   `E3`   |      `F3`       |
|   7   |    1    |      `A4`       |   `B4`   |      `C4`       |
|   8   |    1    |      `D4`       |   `E4`   |      `F4`       |
|   9   |    1    |      `A5`       |   `B5`   |      `C5`       |
|  10   |    1    |      `D5`       |   `E5`   |      `F5`       |

Notes:
- Each chain represents a replication path for a file chunk.
- Targets are storage locations, typically on SSDs.
- CRAQ (Chain Replication with Apportioned Queries) is used for write propagation.
- Read requests can be sent to any target in a chain for load balancing.
```

--------------------------------

### 3fs Example Usage

Source: https://github.com/deepseek-ai/3fs/blob/main/src/lib/api/UsrbIo.md

An example demonstrating the usage of the 3fs library for performing read operations. It includes creating an I/O request manager, an I/O vector, registering a file descriptor, preparing and submitting read requests, waiting for completion, and cleaning up resources.

```c
#include <hf3fs_usrbio.h>
#include <fcntl.h>
#include <unistd.h>

constexpr uint64_t NUM_IOS = 1024;
constexpr uint64_t BLOCK_SIZE = (32 << 20);

int main() {
    struct hf3fs_ior ior;
    hf3fs_iorcreate4(&ior, "/hf3fs/mount/point", NUM_IOS, true, 0, 0, -1, 0);
    
    struct hf3fs_iov iov;
    hf3fs_iovcreate(&iov, "/hf3fs/mount/point", NUM_IOS * BLOCK_SIZE, 0, -1);

    int fd = open("/hf3fs/mount/point/example.bin", O_RDONLY);
    hf3fs_reg_fd(fd, 0);

    for (int i = 0; i < NUM_IOS; i++) {
        hf3fs_prep_io(&ior, &iov, true, iov.base + i * BLOCK_SIZE, fd, i * BLOCK_SIZE, BLOCK_SIZE, nullptr);
    }
    hf3fs_submit_ios(&ior);

    hf3fs_cqe cqes[NUM_IOS];
    hf3fs_wait_for_ios(&ior, cqes, NUM_IOS, NUM_IOS, nullptr);

    hf3fs_dereg_fd(fd);
    close(fd);
    hf3fs_iovdestroy(&iov);
    hf3fs_iordestroy(&ior);

    return 0;
}

```

--------------------------------

### Generate Data Placement Solution

Source: https://github.com/deepseek-ai/3fs/blob/main/deploy/data_placement/README.md

Executes the data placement problem solver script. This script uses Pyomo and HiGHS to find an optimal solution based on specified cluster parameters. The output is an incidence matrix used for generating the chain table.

```bash
cd deploy/data_placement
python src/model/data_placement.py -ql -relax -type CR --num_nodes 5 --replication_factor 3 --min_targets_per_disk 6 --init_timelimit 600
```

--------------------------------

### pybind11 Module Setup and Build Configuration

Source: https://github.com/deepseek-ai/3fs/blob/main/src/lib/py/CMakeLists.txt

This snippet configures the pybind11 module for the 3fs project, specifying the source file and target name. It also sets up compile definitions, include directories, and links the necessary shared library for the project.

```cmake
pybind11_add_module(hf3fs_py_usrbio usrbio_binding.cc)

target_compile_definitions(hf3fs_py_usrbio
                           PRIVATE) # VERSION_INFO=${PYCLIENT_VERSION_INFO})

target_include_directories(hf3fs_py_usrbio
  PUBLIC
  $<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/src>
  ${PROJECT_SOURCE_DIR}
  ${PROJECT_BINARY_DIR}/src
  ${PROJECT_BINARY_DIR}
  )
target_link_libraries(hf3fs_py_usrbio PRIVATE hf3fs_api_shared)
```

--------------------------------

### Add Binary Target

Source: https://github.com/deepseek-ai/3fs/blob/main/src/simple_example/CMakeLists.txt

Defines an executable target named 'simple_example_main' which is built from the source file 'main.cpp' and depends on the library target 'simple_example'. This specifies the main entry point for the application.

```cmake
target_add_bin(simple_example_main "main.cpp" simple_example)
```

--------------------------------

### Add Library Target

Source: https://github.com/deepseek-ai/3fs/blob/main/src/simple_example/CMakeLists.txt

Defines a library target named 'simple_example' and links it with several other libraries including 'core-app', 'core-user', 'core-service', 'fdb', 'simple_example-fbs', 'mgmtd-client', 'storage-client', 'memory-common', and 'analytics'. This is a common pattern for managing dependencies in a C++ project using a build system like CMake.

```cmake
target_add_lib(simple_example core-app core-user core-service fdb simple_example-fbs mgmtd-client storage-client memory-common analytics)
```

--------------------------------

### SCN Library Build Options

Source: https://github.com/deepseek-ai/3fs/blob/main/CMakeLists.txt

This configures the build options for the scnlib library, disabling tests, examples, benchmarks, documentation, installation, and pedantic checks. This is typically done to reduce build times or dependencies for specific build configurations.

```cmake
set(SCN_TESTS OFF)
set(SCN_EXAMPLES OFF)
set(SCN_BENCHMARKS OFF)
set(SCN_DOCS OFF)
set(SCN_INSTALL OFF)
set(SCN_PEDANTIC OFF)
```

--------------------------------

### Automate Service Creation with Bash Script

Source: https://github.com/deepseek-ai/3fs/blob/main/src/simple_example/README.md

This bash script automates the process of creating a new service by copying existing service files, renaming them, and performing find-and-replace operations for service names within the copied files. It ensures that both the main service files and FlatBuffers definitions are correctly duplicated and updated.

```bash
svr_name='migration'
SrvName='Migration'
mkdir -p "src/$svr_name" && pushd src/simple_example && cp -rf --parents . "../$svr_name" && popd
mkdir -p "src/fbs/$svr_name" && pushd src/fbs/simple_example && cp -rf --parents . "../$svr_name" && popd
find "src/$svr_name" "src/fbs/$svr_name" -type f | xargs sed -i "s/simple_example/$svr_name/g"
find "src/$svr_name" "src/fbs/$svr_name" -type f | xargs sed -i "s/SimpleExample/$SrvName/g"
```

--------------------------------

### Add Binary Target for Migration Main

Source: https://github.com/deepseek-ai/3fs/blob/main/src/migration/CMakeLists.txt

This command defines a binary target named 'migration_main'. It specifies the source file 'main.cpp' and links it to the 'migration' library. This is the main executable for the migration process.

```cpp
target_add_bin(migration_main "main.cpp" migration)
```

--------------------------------

### Build 3FS using CMake

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Configures and builds the 3FS project using CMake. It specifies the C++ and C compilers, build type, and enables compilation command export.

```bash
cmake -S . -B build -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cmake --build build -j 32
```

--------------------------------

### CRAQ Failure and Recovery Example

Source: https://github.com/deepseek-ai/3fs/blob/main/docs/design_notes.md

Illustrates a scenario where a successor node fails during a write request, and how the system recovers by updating the chain and retrying the request with a new successor.

```text
Scenario: A, B, C are targets. Write request at A forwards to B. B fails. Cluster manager marks B offline, moves to end. A receives updated table, forwards to C. C may reject initially but eventually accepts.
```

--------------------------------

### Timer Component Origin

Source: https://github.com/deepseek-ai/3fs/blob/main/specs/README.md

Information about the Timer component, noting that it includes modified portions of an open-source project from P tutorials, licensed under the MIT License.

```Markdown
The [`Timer`](Timer) includes modified portions of the following open-source project:
  - The [original implementation](https://github.com/p-org/P/tree/master/Tutorial/Common/Timer) of `Timer` is part of [P tutorials](https://p-org.github.io/P/tutsoutline/) licensed under MIT License.
```

--------------------------------

### Add Common Client Libraries

Source: https://github.com/deepseek-ai/3fs/blob/main/src/lib/common/CMakeLists.txt

This command adds common client libraries to the project, including 'common', 'numa', and 'rt'. It is typically used during the build or setup phase of the project.

```shell
target_add_lib(client-lib-common common numa rt)
```

--------------------------------

### Storage Target State Transition Table

Source: https://github.com/deepseek-ai/3fs/blob/main/docs/design_notes.md

Illustrates the state transitions for storage targets based on their local state, current public state, and predecessor's public state. This table guides the cluster manager in updating public states.

```APIDOC
State Transition Logic:

Local State: up-to-date
  Current Public State: serving -> Next Public State: serving
  Current Public State: syncing -> Next Public State: serving
  Current Public State: waiting -> Next Public State: waiting
  Current Public State: lastsrv -> Next Public State: serving
  Current Public State: offline -> Next Public State: waiting

Local State: online
  Current Public State: serving -> Next Public State: serving
  Current Public State: syncing, Predecessor's Public State: serving -> Next Public State: syncing
  Current Public State: syncing, Predecessor's Public State: not serving -> Next Public State: waiting
  Current Public State: waiting, Predecessor's Public State: serving -> Next Public State: syncing
  Current Public State: waiting, Predecessor's Public State: not serving -> Next Public State: waiting
  Current Public State: lastsrv -> Next Public State: serving
  Current Public State: offline -> Next Public State: waiting

Local State: offline
  Current Public State: serving, Predecessor's Public State: has no predecessor -> Next Public State: lastsrv
  Current Public State: serving, Predecessor's Public State: has predecessor -> Next Public State: offline
  Current Public State: syncing -> Next Public State: offline
  Current Public State: waiting -> Next Public State: offline
  Current Public State: lastsrv -> Next Public State: lastsrv
  Current Public State: offline -> Next Public State: offline
```

--------------------------------

### Build 3FS using Docker for TencentOS-4

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Pulls the Docker image for building 3FS on TencentOS-4.

```dockerfile
docker pull docker.io/tencentos/tencentos4-deepseek3fs-build:latest
```

--------------------------------

### Optimizing for Batched Small I/Os

Source: https://github.com/deepseek-ai/3fs/blob/main/benchmarks/fio_usrbio/README.md

Configuration parameters for benchmarking batched small I/Os using the FIO engine. Sets iodepth and related batch parameters to a high value.

```fio
iodepth=1024
iodepth_batch_submit=1024
iodepth_batch_complete_min=1024
iodepth_batch_complete_max=1024
```

--------------------------------

### Project Subdirectories

Source: https://github.com/deepseek-ai/3fs/blob/main/src/fbs/CMakeLists.txt

This snippet lists the main subdirectories within the Deepseek AI 3fs project. Each subdirectory typically represents a distinct module or component of the project, such as core functionalities, management daemons, metadata services, storage components, monitoring collectors, examples, and migration tools.

```cmake
add_subdirectory(core)
add_subdirectory(mgmtd)
add_subdirectory(meta)
add_subdirectory(storage)
#add_subdirectory(lib)
add_subdirectory(monitor_collector)
add_subdirectory(simple_example)
add_subdirectory(migration)
```

--------------------------------

### Build 3FS using Docker for OpenCloudOS-9

Source: https://github.com/deepseek-ai/3fs/blob/main/README.md

Pulls the Docker image for building 3FS on OpenCloudOS-9.

```dockerfile
docker pull docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest
```

--------------------------------

### Add Libraries to Migration Target

Source: https://github.com/deepseek-ai/3fs/blob/main/src/migration/CMakeLists.txt

This command adds several core libraries to the 'migration' build target. These libraries include 'core-app', 'core-user', 'core-service', 'fdb', 'migration-fbs', 'mgmtd-client', 'storage-client', 'memory-common', and 'analytics'. These are essential dependencies for the migration functionality.

```cpp
target_add_lib(migration core-app core-user core-service fdb migration-fbs mgmtd-client storage-client memory-common analytics)
```

--------------------------------

### Build FIO Engine for 3FS USRBIO

Source: https://github.com/deepseek-ai/3fs/blob/main/benchmarks/fio_usrbio/README.md

Builds the FIO external plugin for 3FS USRBIO. Requires setting environment variables for 3FS and FIO directories. The output is the 'hf3fs_usrbio.so' plugin.

```bash
make HF3FS_LIB_DIR=${HF3FS_LIB_DIR} HF3FS_INCLUDE_DIR=${HF3FS_INCLUDE_DIR} FIO_SRC_DIR=${FIO_SRC_DIR}
```

--------------------------------

### Add Libraries to Target

Source: https://github.com/deepseek-ai/3fs/blob/main/src/storage/CMakeLists.txt

Configures the build target 'storage' to include a list of libraries such as 'core-app', 'core-service', 'memory-common', 'storage-fbs', 'mgmtd-client', 'storage-client', 'kv', 'analytics', 'aio', 'chunk_engine', and 'profiler'. This ensures all necessary components are linked.

```cmake
target_add_lib(storage core-app core-service memory-common storage-fbs mgmtd-client storage-client kv analytics aio chunk_engine profiler)
```

--------------------------------

### Usage of FIO Engine with 3FS

Source: https://github.com/deepseek-ai/3fs/blob/main/benchmarks/fio_usrbio/README.md

Demonstrates how to use the 'hf3fs_usrbio.so' FIO plugin. Specifies the ioengine and mountpoint in the FIO configuration.

```fio
ioengine=external:hf3fs_usrbio.so
mountpoint=/hf3fs/mount/point
```

--------------------------------

### trash_cleaner CLI Usage

Source: https://github.com/deepseek-ai/3fs/blob/main/src/client/trash_cleaner/README.md

Provides the command-line interface usage instructions for the trash_cleaner tool. It outlines the required options and available flags for controlling its behavior, such as scan interval, logging, and target directories.

```bash
trash_cleaner 0.1.0

USAGE:
    trash_cleaner [FLAGS] [OPTIONS] --interval <interval>

FLAGS:
        --abort-on-error    
    -h, --help              Prints help information
    -V, --version           Prints version information

OPTIONS:
    -i, --interval <interval>            Scan interval (in seconds), exit after one scan if set to 0
        --log <log>                      Log path, default is current directory [default: ./]
        --log-level <log-level>          Log level, default is info [default: info]
    -p, --paths <paths>...               path to trash directory
        --stdout-level <stdout-level>    stdout log level, default is warn [default: warn]
```

--------------------------------

### Add Binary to Build System

Source: https://github.com/deepseek-ai/3fs/blob/main/src/client/bin/CMakeLists.txt

This command adds a binary target to the build system. It specifies the target name, the source file, and the output binary name.

```bash
target_add_bin(admin_cli "admin_cli.cc" admin-cli)
```

--------------------------------

### Main Source and Test Directories

Source: https://github.com/deepseek-ai/3fs/blob/main/CMakeLists.txt

This snippet adds the main source, tests, and benchmarks directories to the build process. This ensures that the project's core logic, unit tests, and performance benchmarks are compiled and included in the build.

```cmake
add_subdirectory(src)
add_subdirectory(tests)
add_subdirectory(benchmarks)
```

--------------------------------

### Project-Wide Settings and Package Finding

Source: https://github.com/deepseek-ai/3fs/blob/main/CMakeLists.txt

This snippet sets project-wide configurations such as Boost static library usage, the project version, and enables testing. It also finds required packages like Threads and Boost, and locates the libuv library.

```cmake
set(Boost_USE_STATIC_LIBS ON)
set(FDB_VERSION 7.1.5-ibe)

find_package(Threads REQUIRED)
find_package(Boost REQUIRED COMPONENTS filesystem system program_options)
find_library(LIBUV_LIBRARY NAMES libuv1)
enable_testing()
```