### Prepare and Mount JuiceFS Source: https://github.com/juicedata/juicefs/blob/main/sdk/python/examples/ffrecord/readme.md Before running the FFRecord examples, ensure JuiceFS is installed and mounted to a local directory. This step prepares the environment for data operations. ```bash # This is a ffrecord dataloader example. # Prepare # Install ffrecord here: https://github.com/HFAiLab/ffrecord # Mount JuiceFS juicefs mount redis://localhost /tmp/jfs -d ``` -------------------------------- ### Start Redis Service with Homebrew Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/mount_at_boot.md If Redis is installed via Homebrew, use this command to start it automatically at boot. ```bash brew services start redis ``` -------------------------------- ### Install and Run Thumbnailer Source: https://github.com/juicedata/juicefs/blob/main/docs/en/guide/gateway.md Clone the Thumbnailer repository, install its dependencies using npm, and start the service. Thumbnailer listens for gateway notifications to generate thumbnails. ```Shell git clone https://github.com/minio/thumbnailer/ npm install ``` ```Shell NODE_ENV=webhook node thumbnail-webhook.js ``` -------------------------------- ### Compile and Install LTP Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/posix_compatibility.md Steps to download, compile, and install the LTP test suite from its source archive. ```bash tar -xvf ltp-full-20260529.tar.xz cd ltp-full-20260529 ./configure make all make install ``` -------------------------------- ### Install JuiceFS Client Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/digitalocean.md Installs the latest JuiceFS client on Ubuntu Server 20.04. Verify installation by checking the command help. ```shell curl -sSL https://d.juicefs.com/install | sh - ``` ```shell $ juicefs NAME: juicefs - A POSIX file system built on Redis and object storage. USAGE: juicefs [global options] command [command options] [arguments...] VERSION: 0.16.2 (2021-08-25T04:01:15Z 29d6fee) COMMANDS: format format a volume mount mount a volume umount unmount a volume gateway S3-compatible gateway sync sync between two storage rmr remove directories recursively info show internal information for paths or inodes bench run benchmark to read/write/stat big/small files gc collect any leaked objects fsck Check consistency of file system profile analyze access log stats show runtime stats status show status of JuiceFS warmup build cache for target directories/files dump dump metadata into a JSON file load load metadata from a previously dumped JSON file help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --verbose, --debug, -v enable debug log (default: false) --quiet, -q only warning and errors (default: false) --trace enable trace log (default: false) --no-agent disable pprof (:6060) agent (default: false) --help, -h show help (default: false) --version, -V print only the version (default: false) COPYRIGHT: Apache License 2.0 ``` -------------------------------- ### Install JuiceFS Client Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/standalone.md Installs the JuiceFS client using a one-click script. This is the recommended method for Linux and macOS. ```shell curl -sSL https://d.juicefs.com/install | sh - ``` -------------------------------- ### Mount JuiceFS and Download Data Source: https://github.com/juicedata/juicefs/blob/main/sdk/python/examples/fsspec/readme.md Demonstrates mounting JuiceFS and downloading a CSV file using wget. This setup is required before running the Python example. ```bash # This example demonstrates how to use the fsspec library to read a CSV file. juicefs mount redis://localhost /tmp/jfs -d # Download the data file wget https://gender-pay-gap.service.gov.uk/viewing/download-data/2021 -O /tmp/jfs/ray_demo_data.csv # run the example python3 sdk/python/examples/fsspec/main.py ``` -------------------------------- ### Install and Mount JuiceFS with Ansible Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/automation.md Use this Ansible playbook to automate the installation and mounting of JuiceFS on a target host. Ensure the JuiceFS client package URL and metadata URL are updated to match your environment. This example assumes a local SQLite database for metadata. ```yaml - hosts: localhost tasks: - set_fact: # Change accordingly meta_url: sqlite3:///tmp/myjfs.db jfs_path: /jfs jfs_pkg: /tmp/juicefs-ce.tar.gz jfs_bin_dir: /usr/local/bin - get_url: # Change download URL accordingly url: https://d.juicefs.com/juicefs/releases/download/v1.0.2/juicefs-1.0.2-linux-amd64.tar.gz dest: "{{jfs_pkg}}" - ansible.builtin.unarchive: src: "{{jfs_pkg}}" dest: "{{jfs_bin_dir}}" include: - juicefs - name: Create symbolic for fstab ansible.builtin.file: src: "{{jfs_bin_dir}}/juicefs" dest: "/sbin/mount.juicefs" state: link - name: Mount JuiceFS and create fstab entry mount: path: "{{jfs_path}}" src: "{{meta_url}}" fstype: juicefs opts: _netdev state: mounted ``` -------------------------------- ### Install FoundationDB Client (RHEL) Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_metadata_engine.md Installs the FoundationDB client library on RHEL-based systems using an .rpm package. ```shell curl -O https://github.com/apple/foundationdb/releases/download/6.3.25/foundationdb-clients-6.3.25-1.el7.x86_64.rpm sudo rpm -Uvh foundationdb-clients-6.3.25-1.el7.x86_64.rpm ``` -------------------------------- ### Install FoundationDB Client (Debian) Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_metadata_engine.md Installs the FoundationDB client library on Debian-based systems using a .deb package. ```shell curl -O https://github.com/apple/foundationdb/releases/download/6.3.25/foundationdb-clients_6.3.25-1_amd64.deb sudo dpkg -i foundationdb-clients_6.3.25-1_amd64.deb ``` -------------------------------- ### Install Samba on Debian/Ubuntu Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/samba.md Installs the Samba server package on Debian-based systems using apt. ```shell sudo apt install samba ``` -------------------------------- ### Install K3s Server Node Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/juicefs_on_k3s.md Use this command to install the K3s server component on a Linux distribution. The script handles service startup and tool installation. ```shell curl -sfL https://get.k3s.io | sh - ``` -------------------------------- ### Load and Start JuiceFS Mount Service Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/mount_at_boot.md Use these commands to load the launch agent configuration and start the JuiceFS mount. Verify the mount point afterwards. ```bash launchctl load ~/Library/LaunchAgents/io.juicefs..plist launchctl start ~/Library/LaunchAgents/io.juicefs. ls ``` -------------------------------- ### Install JuiceFS via Snapcraft Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Install the JuiceFS client using Snapcraft on Ubuntu 16.04 and above, or other Snap-supported systems. ```shell sudo snap install juicefs ``` -------------------------------- ### Install JuiceFS Client to System Path Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Installs the extracted JuiceFS client binary to /usr/local/bin, making it available system-wide. Requires sudo privileges. ```shell sudo install juicefs /usr/local/bin ``` -------------------------------- ### Install liburing Source: https://github.com/juicedata/juicefs/blob/main/test/io_uring_test/readme_en.md Install the liburing library, which is required for io_uring operations. Instructions are provided for Ubuntu/Debian and CentOS/RHEL, as well as building from source. ```bash # Ubuntu/Debian sudo apt install liburing-dev # CentOS/RHEL sudo yum install liburing-devel # Or build from source git clone https://github.com/axboe/liburing.git cd liburing && make && sudo make install ``` -------------------------------- ### Install JuiceFS Client via APT Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Installs the JuiceFS client package from the configured PPA repository on Ubuntu systems. ```shell sudo apt-get install juicefs ``` -------------------------------- ### Install and Enable Netmount Service on Alpine Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/troubleshooting.md Install the 'netmount' package and enable the service for automatic network file system mounting on Alpine Linux. ```bash # use --update-fstab to add juicefs mount to /etc/fstab # install and enable netmount service apk add openrc rc-update add netmount boot # * service netmount added to runlevel boot rc-service netmount start # / # rc-service netmount start # * Mounting network filesystems ... ``` -------------------------------- ### Install WSL Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/juicefs_on_wsl.md Installs the Windows Subsystem for Linux. This command downloads the Linux kernel, sets WSL 2 as default, and installs Ubuntu by default. ```powershell wsl --install ``` -------------------------------- ### Install Samba on RHEL/CentOS/Fedora Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/samba.md Installs the Samba server package on RHEL-based systems using dnf. ```shell sudo dnf install samba ``` -------------------------------- ### Enable and start JuiceFS mount with systemd Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/mount_at_boot.md After creating the systemd.mount file, create the necessary symlink for mount.juicefs and then enable and start the systemd service for JuiceFS. This ensures the JuiceFS instance is mounted on boot. ```bash ln -s /usr/local/bin/juicefs /sbin/mount.juicefs systemctl enable juicefs.mount systemctl start juicefs.mount ``` -------------------------------- ### Install fsspec Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/python_sdk.md Install the fsspec library to enable JuiceFS to work with fsspec-compatible frameworks. ```bash pip install fsspec ``` -------------------------------- ### Multiple Mount Options Example Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/mount_at_boot.md If you have multiple mount options, list them as separate string elements within the ProgramArguments array. ```xml --max-uploads 50 --cache-size 204800 ``` -------------------------------- ### Install JuiceFS on Windows via Scoop Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Install the latest version of the JuiceFS client on Windows using the Scoop package manager. ```shell scoop install juicefs ``` -------------------------------- ### Install JuiceFS Python SDK Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/python_sdk.md Install the compiled JuiceFS Python SDK using pip. Ensure the .whl file is on the target machine. ```bash pip install juicefs-1.3.0-py3-none-any.whl ``` -------------------------------- ### Start JuiceFS WebDAV Server Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/command_reference.mdx Starts a WebDAV server. This command requires a metadata URL and an address for the WebDAV server to listen on. Optional flags can be used for HTTPS, compression, and other configurations. ```shell juicefs webdav redis://localhost localhost:9007 ``` -------------------------------- ### Start JuiceFS S3 Gateway with Metrics Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/monitoring.md Start the JuiceFS S3 Gateway and expose monitoring metrics on a specified address. Ensure JuiceFS client version 0.17.1 or above is used. ```shell juicefs gateway --metrics localhost:9567 ... ``` -------------------------------- ### Compile and Install JuiceFS from AUR on Arch Linux Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Manually compile and install the JuiceFS client from the AUR on Arch Linux by cloning the repository and using makepkg. ```shell sudo pacman -S base-devel git go ``` ```shell git clone https://aur.archlinux.org/juicefs.git ``` ```shell cd juicefs ``` ```shell makepkg -si ``` -------------------------------- ### Start Spark Thrift Server Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/hadoop_java_sdk.md Starts the Spark Thrift JDBC/ODBC Server with specified configurations for memory, cores, and Spark properties. Ensure SPARK_HOME is set. ```shell ${SPARK_HOME}/sbin/start-thriftserver.sh \ --master yarn \ --driver-memory 8g \ --executor-memory 10g \ --executor-cores 3 \ --num-executors 3 \ --conf spark.locality.wait=100 \ --conf spark.sql.crossJoin.enabled=true \ --hiveconf hive.server2.thrift.port=10001 ``` -------------------------------- ### Common JuiceFS Hadoop Java SDK Configuration Example Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/hadoop_java_sdk.md A typical configuration for JuiceFS in a Hadoop environment. Replace placeholders like {HOST}, {PORT}, and {DB} with your specific details. This example sets up the file system implementation, meta-store connection, cache directory, cache size, and access log path. ```xml fs.jfs.impl io.juicefs.JuiceFileSystem fs.AbstractFileSystem.jfs.impl io.juicefs.JuiceFS juicefs.meta redis://{HOST}:{PORT}/{DB} juicefs.cache-dir /data*/jfs juicefs.cache-size 1024 juicefs.access-log /tmp/juicefs.access.log ``` -------------------------------- ### Install JuiceFS on Arch Linux via Yay Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Install the JuiceFS client on Arch Linux and its derivatives using the Yay package manager. ```shell yay -S juicefs ``` -------------------------------- ### Start Authenticated WebDAV Server Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/webdav.md To enable authentication, set the WEBDAV_USER and WEBDAV_PASSWORD environment variables before starting the server. This requires JuiceFS v1.0.3 or later. ```shell export WEBDAV_USER=user export WEBDAV_PASSWORD=mypassword sudo juicefs webdav sqlite3://myjfs.db 192.168.1.8:80 ``` -------------------------------- ### Install Rancher on Docker Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/juicefs_on_rancher.md Run this command to install Rancher on a Linux system using Docker. Ensure Docker is version 19.03 or higher and the host has at least 4GB of memory. ```shell sudo docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher ``` -------------------------------- ### Install JuiceFS Client in Dockerfile Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/how_to_use_on_kubernetes.md Installs the JuiceFS client by downloading the latest release, extracting it, and placing it in the PATH. This snippet is intended for use within a Dockerfile. ```dockerfile RUN apk add --no-cache curl && \ JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v') && \ wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ install juicefs /usr/bin && \ rm juicefs "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ rm -rf /var/cache/apk/* && \ apk del curl ENTRYPOINT ["/usr/bin/juicefs", "mount"] ``` -------------------------------- ### Start Anonymous WebDAV Server Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/webdav.md Use this command to start an anonymous WebDAV server for JuiceFS. Replace META-URL with your JuiceFS metadata URL and LISTENING-ADDRESS:PORT with the desired network interface and port. ```shell juicefs webdav META-URL LISTENING-ADDRESS:PORT ``` ```shell sudo juicefs webdav sqlite3://myjfs.db 192.168.1.8:80 ``` -------------------------------- ### Start JuiceFS S3 Gateway Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/command_reference.mdx Starts an S3-compatible gateway. Ensure to set environment variables for user and password if required. This command requires a metadata URL and an address for the gateway to listen on. ```shell export MINIO_ROOT_USER=admin export MINIO_ROOT_PASSWORD=12345678 juicefs gateway redis://localhost localhost:9000 ``` -------------------------------- ### File Data Chunking Example Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/development/internals.md Illustrates how a large file is divided into chunks, with each chunk potentially containing multiple slices. ```text File: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _| Chunk: |<--- Chunk 0 --->|<--- Chunk 1 --->|<-- Chunk 2 -->| ``` -------------------------------- ### Set up Cloud SQL Proxy Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/juicefs_on_colab.md Download, make executable, and start the Cloud SQL proxy. This allows JuiceFS to connect to the Cloud SQL instance. ```shell ! wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy ! chmod +x cloud_sql_proxy ! GOOGLE_APPLICATION_CREDENTIALS=/content/.config/application_default_credentials.json nohup ./cloud_sql_proxy -instances=juicefs-learning:europe-west1:juicefs-sql-example-1=tcp:0.0.0.0:5432 >> cloud_sql_proxy.log & ``` -------------------------------- ### Mount JuiceFS with etcd Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_metadata_engine.md Example command to mount a JuiceFS file system using etcd as the metadata engine. ```shell juicefs mount -d "etcd://192.168.1.6:2379,192.168.1.7:2379,192.168.1.8:2379/jfs" /mnt/jfs ``` -------------------------------- ### Install JuiceFS Client with Homebrew on macOS Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Use this command if you have Homebrew installed on your macOS system to install the JuiceFS client. ```shell brew install juicefs ``` -------------------------------- ### Install Specific WSL Distribution Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/juicefs_on_wsl.md Installs a specific Linux distribution, such as Ubuntu, on WSL. ```powershell wsl --install -d ubuntu ``` -------------------------------- ### Build JuiceFS Binaries Source: https://github.com/juicedata/juicefs/blob/main/AGENTS.md Use these commands to build the JuiceFS binary with different build tags and configurations. The standard build produces `./juicefs`. ```shell make juicefs # standard build -> ./juicefs STATIC=1 make juicefs # static binary (needs musl-gcc) make BUILD=debug all # debug build (-N -l) make juicefs.lite # minimal build, most backends disabled make juicefs.ceph # -tags ceph make juicefs.fdb # -tags fdb (FoundationDB) ``` -------------------------------- ### Verify JuiceFS Installation Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/windows.md Run this command in your terminal to check if the JuiceFS client is installed correctly and to see its version. ```bash juicefs version ``` -------------------------------- ### Instantiate JuiceFS Client Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/python_sdk.md Create a JuiceFS client instance using the Client class. Provide the file system name and metadata engine URL. ```python from juicefs import Client # Create JuiceFS client jfs = Client(name='', meta='redis://192.168.1.8/0') # List files in a directory jfs.listdir('/') ``` -------------------------------- ### Install JuiceFS on Fedora via Copr Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Enable the Copr repository and install the JuiceFS client on Fedora and its derivatives. ```shell sudo dnf copr enable -y juicedata/juicefs ``` ```shell sudo dnf install juicefs ``` -------------------------------- ### Data Distribution Example Source: https://github.com/juicedata/juicefs/blob/main/docs/en/development/internals.md Shows the actual data distribution for a 64 MiB chunk, detailing how different parts of the data map to specific objects in object storage. This helps understand data fragmentation and object mapping. ```text 0 ~ 10M: Zero 10 ~ 16M: 10_0_4194304, 10_1_4194304(0 ~ 2M) 16 ~ 26M: 12_0_4194304, 12_1_4194304, 12_2_2097152 26 ~ 36M: 11_1_4194304(2 ~ 4M), 11_2_4194304, 11_3_4194304 36 ~ 40M: 10_6_4194304(2 ~ 4M), 10_7_2097152 40 ~ 64M: Zero ``` -------------------------------- ### Install libglusterfs-dev on Debian Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_object_storage.md Installs necessary development libraries for GlusterFS on Debian-based systems. This includes uuid-dev, libglusterfs-dev, and glusterfs-common. ```bash sudo apt-get install uuid-dev libglusterfs-dev glusterfs-common ``` -------------------------------- ### Install librados2-devel on CentOS Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_object_storage.md Installs the librados development package on RHEL-based systems. This is required for JuiceFS to interact with Ceph RADOS. ```bash sudo yum install librados2-devel ``` -------------------------------- ### Install librados-dev on Debian Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_object_storage.md Installs the librados development package on Debian-based systems. This is required for JuiceFS to interact with Ceph RADOS. ```bash sudo apt-get install librados-dev ``` -------------------------------- ### Format a volume with quota and capacity limits Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/command_reference.mdx This example illustrates creating a JuiceFS volume with specific limits on the number of inodes and total capacity. ```shell juicefs format sqlite3://myjfs.db myjfs --inodes=1000000 --capacity=102400 ``` -------------------------------- ### Install kirk Test Runner Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/posix_compatibility.md Installs the kirk test runner, which replaces the older runltp for executing LTP tests. ```bash pip install kirk ``` -------------------------------- ### Install musl-cross for macOS Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Install the musl-cross toolchain on macOS using Homebrew. This is a dependency for cross-compiling Linux clients on macOS. ```shell brew install FiloSottile/musl-cross/musl-cross ``` -------------------------------- ### Install MinGW-w64 on Ubuntu Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Install the MinGW-w64 cross-compilation toolchain on Ubuntu systems. This is required for cross-compiling Windows clients on Linux. ```shell sudo apt install mingw-w64 ``` -------------------------------- ### JuiceFS CLI Help Source: https://github.com/juicedata/juicefs/blob/main/docs/en/tutorials/aliyun.md Displays the help message for the JuiceFS command-line interface, showing available commands and global options. This output confirms successful installation. ```shell $ juicefs NAME: juicefs - A POSIX file system built on Redis and object storage. USAGE: juicefs [global options] command [command options] [arguments...] VERSION: 0.15.2 (2021-07-07T05:51:36Z 4c16847) COMMANDS: format format a volume mount mount a volume umount unmount a volume gateway S3-compatible gateway sync sync between two storage rmr remove directories recursively info show internal information for paths or inodes bench run benchmark to read/write/stat big/small files gc collect any leaked objects fsck Check consistency of file system profile analyze access log status show status of JuiceFS warmup build cache for target directories/files dump dump metadata into a JSON file load load metadata from a previously dumped JSON file help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --verbose, --debug, -v enable debug log (default: false) --quiet, -q only warning and errors (default: false) --trace enable trace log (default: false) --no-agent disable pprof (:6060) agent (default: false) --help, -h show help (default: false) --version, -V print only the version (default: false) COPYRIGHT: Apache License 2.0 ``` -------------------------------- ### 调整 JuiceFS 内存占用相关配置 Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/administration/troubleshooting.md 通过调整读写缓冲区大小和 Go Runtime 的 GC 策略来优化 JuiceFS 客户端的内存占用。注意调整可能影响性能。 ```shell # 降低读写缓冲区大小 juicefs mount -o buffer-size=1MiB ... ``` ```shell # 降低 GOGC 阈值,例如设置为 50 export GOGC=50 juicefs mount ... ``` -------------------------------- ### POST Request Example for AssumeRole Source: https://github.com/juicedata/juicefs/blob/main/docs/en/guide/gateway.md This is an example of a POST request to the AssumeRole API, typically used for obtaining temporary credentials. ```http http://minio:9000/?Action=AssumeRole&DurationSeconds=3600&Version=2011-06-15&Policy={"Version":"2012-10-17","Statement":[{"Sid":"Stmt1","Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}]}&AUTHPARAMS ``` -------------------------------- ### Install GlusterFS packages on CentOS Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_object_storage.md Installs GlusterFS runtime and development packages on RHEL-based systems. This includes glusterfs, glusterfs-api-devel, and glusterfs-libs. ```bash sudo yum install glusterfs glusterfs-api-devel glusterfs-libs ``` -------------------------------- ### Start HTTPS WebDAV Server Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/webdav.md Launch the WebDAV server with HTTPS support enabled by providing the paths to your certificate and private key files. The listening address and port should be configured for HTTPS traffic. ```shell sudo juicefs webdav \ --cert-file ./client.crt \ --key-file ./client.key \ sqlite3://myjfs.db 192.168.1.8:443 ``` -------------------------------- ### One-click Install JuiceFS (Custom Location) Source: https://github.com/juicedata/juicefs/blob/main/docs/en/getting-started/installation.md Installs the latest JuiceFS client to a specified custom directory, such as /tmp, on Linux and macOS. ```shell curl -sSL https://d.juicefs.com/install | sh -s /tmp ``` -------------------------------- ### Example JuiceFS Access Log Entry Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/troubleshooting.md An example of a single access log entry showing a read operation with its parameters and timing. ```log 2022.09.22 08:55:21.013121 [uid:0,gid:0,pid:0] read (148153116,131072,28668010496): OK (131072) <1.309992> ``` -------------------------------- ### Start S3 Gateway with Access Log Enabled Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/fault_diagnosis_and_analysis.md Enable access log output for the S3 gateway by adding the `--access-log` option during startup. This helps in monitoring and analyzing S3 requests. ```bash --access-log ``` -------------------------------- ### Install JuiceFS Docker Volume Plugin Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/juicefs_on_docker.md Installs the JuiceFS Docker volume plugin. Ensure you provide necessary permissions for FUSE when prompted. ```shell docker plugin install juicedata/juicefs ``` -------------------------------- ### 在 Docker 镜像中集成 JuiceFS 客户端 Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/deployment/how_to_use_on_kubernetes.md 使用此 Dockerfile 将 JuiceFS 客户端安装到应用镜像中,以便在容器内直接挂载 JuiceFS。该示例使用 Alpine Linux 作为基础镜像。 ```dockerfile FROM alpine:latest LABEL maintainer="Juicedata " # Install JuiceFS client RUN apk add --no-cache curl && \ JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v') && \ wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ install juicefs /usr/bin && \ rm juicefs "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ rm -rf /var/cache/apk/* && \ apk del curl ENTRYPOINT ["/usr/bin/juicefs", "mount"] ``` -------------------------------- ### List Files using MinIO Client Source: https://github.com/juicedata/juicefs/blob/main/docs/en/guide/gateway.md Use the `mc ls` command with the alias to list files and directories within a JuiceFS bucket. ```shell $ mc ls juicefs/jfs [2021-10-20 11:59:00 CST] 130KiB avatar-2191932_1920.png [2021-10-20 11:59:00 CST] 4.9KiB box-1297327.svg [2021-10-20 11:59:00 CST] 21KiB cloud-4273197.svg [2021-10-20 11:59:05 CST] 17KiB hero.svg [2021-10-20 11:59:06 CST] 1.7MiB hugo-rocha-qFpnvZ_j9HU-unsplash.jpg [2021-10-20 11:59:06 CST] 16KiB man-1352025.svg [2021-10-20 11:59:06 CST] 1.3MiB man-1459246.ai [2021-10-20 11:59:08 CST] 19KiB sign-up-accent-left.07ab168.svg [2021-10-20 11:59:10 CST] 11MiB work-4997565.svg ``` -------------------------------- ### Install NFS Client (Ubuntu/Debian) Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/nfs.md Installs the NFS common client package on Linux hosts. This is required on all client machines that will access the NFS shares. ```shell sudo apt install nfs-common ``` -------------------------------- ### Local Benchmark: Create Files Source: https://github.com/juicedata/juicefs/blob/main/docs/en/deployment/hadoop_java_sdk.md Use this command to create a specified number of empty files locally for metadata benchmarking. ```shell hadoop jar juicefs-hadoop.jar nnbench create -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local ``` -------------------------------- ### Sync with Start Time Filter Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/command_reference.mdx Skips files modified before a specified start time. This allows synchronization of files that have been modified on or after a certain date and time. ```shell --start-time 1.3|skip files modified before start-time. example: 2006-01-02 15:04:05 ``` -------------------------------- ### Data Distribution Example for a 64 MiB Chunk Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/development/internals.md Shows the breakdown of a 64 MiB chunk into multiple objects, including padding and data distribution across different slices and blocks. ```text 0 ~ 10M: 补零 10 ~ 16M: 10_0_4194304, 10_1_4194304(0 ~ 2M) 16 ~ 26M: 12_0_4194304, 12_1_4194304, 12_2_2097152 26 ~ 36M: 11_1_4194304(2 ~ 4M), 11_2_4194304, 11_3_4194304 36 ~ 40M: 10_6_4194304(2 ~ 4M), 10_7_2097152 40 ~ 64M: 补零 ``` -------------------------------- ### 诊断目录统计不一致 Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/guide/dir-stats.md 当快速模式和严格模式下的 `juicefs info -r` 命令结果不一致时,可能表示目录统计存在问题。使用 `juicefs fsck` 命令配合 `--path` 和 `--sync-dir-stat` 选项来检查和同步目录统计信息。 ```shell $ juicefs info -r /jfs/d /jfs/d: 1 3.3/s /jfs/d: 448.0 MiB (469766144 Bytes) 1.4 GiB/s /jfs/d : inode: 2 files: 1 dirs: 1 length: 448.00 MiB (469762048 Bytes) size: 448.00 MiB (469766144 Bytes) path: /d $ juicefs info -r --strict /jfs/d /jfs/d: 1 3.3/s /jfs/d: 1.0 GiB (1073745920 Bytes) 3.3 GiB/s /jfs/d : inode: 2 files: 1 dirs: 1 length: 1.00 GiB (1073741824 Bytes) size: 1.00 GiB (1073745920 Bytes) path: /d # 检查目录 /d 的用量统计 $ juicefs fsck sqlite3://test.db --path /d --sync-dir-stat 2023/05/31 17:14:34.700239 juicefs[32667] : Meta address: sqlite3://test.db [interface.go:494] [xorm] [info] 2023/05/31 17:14:34.700291 PING DATABASE sqlite3 2023/05/31 17:14:34.701553 juicefs[32667] : usage stat of /d should be &{1073741824 1073741824 1}, but got &{469762048 469762048 1} [base.go:2010] 2023/05/31 17:14:34.701577 juicefs[32667] : Stat of path /d (inode 2) should be synced, please re-run with '--path /d --repair --sync-dir-stat' to fix it [base.go:2025] 2023/05/31 17:14:34.701615 juicefs[32667] : some errors occurred, please check the log of fsck [main.go:31] ``` -------------------------------- ### Start Prometheus Service Source: https://github.com/juicedata/juicefs/blob/main/docs/en/administration/monitoring.md Run the Prometheus executable with the specified configuration file to start scraping metrics. Access the Prometheus interface at http://localhost:9090. ```shell ./prometheus --config.file=prometheus.yml ``` -------------------------------- ### Redis Cluster Key Prefix Example Source: https://github.com/juicedata/juicefs/blob/main/docs/en/reference/how_to_set_up_metadata_engine.md Demonstrates how to use different database numbers as key prefixes in Redis Cluster for distinct JuiceFS volumes. This ensures all keys for a single volume are routed to the same slot. ```shell juicefs format redis://cluster:6379/1 volume1 juicefs format redis://cluster:6379/2 volume2 ``` -------------------------------- ### Prepare Python SDK and Read with JuiceFS Python SDK Dataloader Source: https://github.com/juicedata/juicefs/blob/main/sdk/python/examples/ffrecord/readme.md This section covers preparing the JuiceFS Python SDK and then using the dataloader script to read the dataset. This demonstrates efficient data loading with the Python SDK. ```bash # Prepare python-sdk make -C sdk/python libjfs ``` ```bash # Read dataset with Juicefs-pythonsdk-dataloader: (takes 10.02s) python3 sdk/python/examples/ffrecord/dataloader.py ``` -------------------------------- ### Apache 2.0 License Header for Go Files Source: https://github.com/juicedata/juicefs/blob/main/AGENTS.md Every new Go file must begin with this Apache 2.0 license header. Ensure the year and copyright holder are correctly updated. ```go /* * JuiceFS, Copyright Juicedata, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ ``` -------------------------------- ### 配置 JuiceFS 挂载服务启动顺序 (systemd) Source: https://github.com/juicedata/juicefs/blob/main/docs/zh_cn/deployment/how_to_use_on_kubernetes.md 配置 systemd unit file,确保 JuiceFS 挂载服务在 Docker 之前启动,以避免容器启动时 JuiceFS 尚未挂载的问题。请根据实际情况确定 JuiceFS 挂载服务的名称。 ```systemd [Unit] # 请使用下方命令确定 JuiceFS 挂载服务的名称(例如 jfs.mount): # systemctl list-units | grep "\.mount" After=network-online.target firewalld.service containerd.service jfs.mount ``` -------------------------------- ### Start redis-cli Monitor Source: https://github.com/juicedata/juicefs/blob/main/docs/en/guide/gateway.md Start the Redis client program and use the `monitor` command to observe all commands executed on Redis in real-time. This is useful for debugging event publishing. ```Shell redis-cli -a yoursecret 127.0.0.1:6379> monitor OK ```