GeeseFS

https://github.com/yandex-cloud/geesefs
Admin
GeeseFS is a high-performance, POSIX-compatible S3 file system written in Go that allows mounting S3...
Tokens:9,156
Snippets:76
Trust Score:7.9
License:Apache-2.0
Update:8 months ago
Show doc for...
Tokens:
Raw
### Build GeeseFS from Source (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

Instructions to clone the GeeseFS repository and build the executable using Go. This method requires Go 1.13 or later to be installed.

```Shell
$ git clone https://github.com/yandex-cloud/geesefs
$ cd geesefs
$ go build
```

--------------------------------

### Mount S3 Bucket with GeeseFS on Windows (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

Example command to mount an S3 bucket using the GeeseFS Windows executable. It shows mounting to a drive letter or a non-existing directory.

```Shell
geesefs-win-x64.exe testbucket K:
```

--------------------------------

### Mount S3 Bucket with GeeseFS (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

Example commands for mounting an S3 bucket using GeeseFS. It shows how to use AWS credentials from a file or environment variables, and how to mount objects under a specific prefix.

```Shell
$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/geesefs <bucket> <mountpoint>
$ $GOPATH/bin/geesefs [--endpoint https://...] <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix
```

--------------------------------

### Configure GeeseFS in /etc/fstab (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

An example entry for the /etc/fstab file to automatically mount an S3 bucket using GeeseFS on system startup. It includes options for file modes and network device mounting.

```Shell
bucket    /mnt/mountpoint    fuse.geesefs    _netdev,allow_other,--file-mode=0666,--dir-mode=0777    0    0
```

--------------------------------

### GeeseFS Go Library Usage

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Integrate GeeseFS into Go applications for programmatic S3 file system access. This example demonstrates creating a GeeseFS instance with custom configuration.

```go
package main

import (
    "context"
    "log"

    "github.com/yandex-cloud/geesefs/core"
    "github.com/yandex-cloud/geesefs/core/cfg"
)

func main() {
    // Create default flags
    flags := cfg.DefaultFlags()
    flags.Endpoint = "https://storage.yandexcloud.net"
    flags.MountPoint = "/mnt/s3"

    // Configure S3 backend
    s3Config := flags.Backend.(*cfg.S3Config)
    s3Config.Region = "ru-central1"

    // Create GeeseFS instance
    ctx := context.Background()
    fs, err := core.NewGoofys(ctx, "mybucket", flags)
    if err != nil {
        log.Fatalf("Failed to create GeeseFS: %v", err)
    }

    // Use fs for operations...
    _ = fs

    // Shutdown when done
    fs.Shutdown()
}
```

--------------------------------

### Mount Azure Blob Storage with Environment Variables (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README-azure.md

Mounts Azure Blob Storage using GeeseFS by setting AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY environment variables. This method is straightforward for quick setups. It requires the geesefs binary to be in the GOPATH/bin directory.

```Shell
$ AZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_KEY=yyy \
    $GOPATH/bin/geesefs wasb://container[:prefix] <mountpoint>
```

--------------------------------

### GeeseFS Custom Multipart Upload Part Sizes

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configures custom multipart upload part sizes for GeeseFS, allowing optimization for different file size requirements. Examples show default, larger parts, and mixed configurations.

```bash
# Default: 1000x5MB + 1000x25MB + 8000x125MB = 1.03TB max
geesefs --part-sizes "5:1000,25:1000,125" mybucket /mnt/s3

# Larger parts for bigger files (10000x50MB = 500GB max)
geesefs --part-sizes "50" mybucket /mnt/s3

# Mixed configuration for varied workloads
geesefs --part-sizes "5:2000,50:4000,100" mybucket /mnt/s3
```

--------------------------------

### GeeseFS High-Throughput Write Configuration

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Optimizes GeeseFS for maximum write throughput by adjusting parameters like disabling checksums, increasing memory limits, and setting higher values for max flushers, parallel parts, and part sizes. Includes an example fio test.

```bash
# Maximum throughput configuration
geesefs \
    --no-checksum \
    --memory-limit 4000 \
    --max-flushers 32 \
    --max-parallel-parts 32 \
    --part-sizes 25 \
    mybucket /mnt/s3

# Test with fio
fio -name=test -ioengine=libaio -direct=1 -bs=4M -iodepth=1 \
    -fallocate=none -numjobs=8 -group_reporting -rw=write -size=10G
```

--------------------------------

### Build API Schema using Go

Source: https://github.com/yandex-cloud/geesefs/blob/master/s3ext/README.md

This snippet demonstrates how to build the API schema for the S3 service using the provided Go tools. It requires the Go build environment and the 'gen-api' tool, which takes the API definition file as input.

```bash
go build -tags=codegen private/model/cli/gen-api/main.go -o gen-api
./gen-api -path service models/apis/s3/2006-03-01/api-2.json
```

--------------------------------

### Persist Changes with fsync in Python

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

Demonstrates how to use the `fsync` system call in Python to ensure that changes made to a file or directory are persisted to the server. This is crucial for data integrity. The function requires a file descriptor obtained from `os.open`.

```python
#!/usr/bin/python

import sys, os
os.fsync(os.open(sys.argv[1], os.O_RDONLY))

```

--------------------------------

### Set AWS Credentials for GeeseFS on Windows (Batch)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

Demonstrates how to set AWS access key ID and secret access key as environment variables in a Windows command prompt for GeeseFS usage.

```Batch
set AWS_ACCESS_KEY_ID=...
set AWS_SECRET_ACCESS_KEY=...
```

--------------------------------

### Benchmark GeeseFS Write Performance with fio

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

This command uses the fio tool to benchmark the write performance of GeeseFS. It configures fio with specific I/O engine, block size, number of jobs, and read/write pattern to simulate a heavy write workload.

```bash
fio -name=test -ioengine=libaio -direct=1 -bs=4M -iodepth=1 -fallocate=none \
    -numjobs=8 -group_reporting -rw=write -size=10G
```

--------------------------------

### Build GeeseFS from Source using Go

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Build the GeeseFS binary from its source code using Go. This includes cloning the repository, building the executable, and running tests.

```bash
# Clone repository
git clone https://github.com/yandex-cloud/geesefs
cd geesefs

# Build binary
go build

# Build with version information
go build -ldflags "-X main.Version=$(git rev-parse HEAD)"

# Install to GOPATH
go install

# Run tests
go test ./core/...
```

--------------------------------

### Basic GeeseFS Mount Command with AWS Credentials

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Demonstrates how to mount an S3 bucket locally using GeeseFS with AWS credentials configured in ~/.aws/credentials. Supports mounting the entire bucket, a specific prefix, or using a custom endpoint.

```bash
# Configure AWS credentials
cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY

# Mount entire bucket
geesefs mybucket /mnt/s3

# Mount with specific prefix
geesefs mybucket:prefix/path /mnt/s3

# Mount with custom endpoint (Amazon S3)
geesefs --endpoint https://s3.amazonaws.com mybucket /mnt/s3

# Mount Yandex Object Storage (default endpoint)
geesefs mybucket /mnt/s3
```

--------------------------------

### Configure GeeseFS for Maximizing Throughput

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

This configuration optimizes GeeseFS for high linear write speeds by increasing parallelism. It involves setting options like `--no-checksum`, `--memory-limit`, `--max-flushers`, `--max-parallel-parts`, and `--part-sizes`. Note that this reduces the maximum file size and increases memory usage.

```bash
geesefs --no-checksum --memory-limit 4000 \
    --max-flushers 32 --max-parallel-parts 32 --part-sizes 25
```

--------------------------------

### Enable Debugging and Logging

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure various debugging options for troubleshooting geesefs. Includes running in foreground with debug output, logging to a file, and enabling the pprof HTTP profiler.

```bash
# Run in foreground with debug output
geesefs -f --debug --debug_fuse --debug_s3 mybucket /mnt/s3

# Log to file
geesefs --log-file /var/log/geesefs.log mybucket /mnt/s3

# Enable pprof HTTP profiler
geesefs --pprof 6060 mybucket /mnt/s3

# Access pprof via HTTP
curl http://localhost:6060/debug/pprof/heap > heap.pprof
go tool pprof heap.pprof
```

--------------------------------

### GeeseFS File Permissions and Ownership Configuration

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Explains how to configure file and directory permissions, ownership (UID/GID), and enable special POSIX features like modification time preservation and special file support in GeeseFS.

```bash
# Set custom permissions
geesefs --file-mode 0644 --dir-mode 0755 mybucket /mnt/s3

# Set custom UID/GID
geesefs --uid 1000 --gid 1000 mybucket /mnt/s3

# Enable POSIX permissions (Yandex S3 only)
geesefs --enable-perms mybucket /mnt/s3

# Enable modification time preservation
geesefs --enable-mtime mybucket /mnt/s3

# Enable special files (sockets, devices, named pipes)
geesefs --enable-specials mybucket /mnt/s3
```

--------------------------------

### GeeseFS Mount with Environment Variable Authentication

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Shows how to authenticate GeeseFS using environment variables for AWS access key ID and secret access key, providing an alternative to the credentials file.

```bash
# Set credentials via environment variables
export AWS_ACCESS_KEY_ID=AKID1234567890
export AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY

# Mount bucket
geesefs mybucket /mnt/s3
```

--------------------------------

### Mount S3 Bucket on Windows using WinFSP

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Mount an S3 bucket on Windows using the WinFSP (Windows File System Proxy). Supports setting credentials via environment variables and mounting to a drive letter or directory.

```batch
REM Set credentials via environment variables
set AWS_ACCESS_KEY_ID=AKID1234567890
set AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY

REM Mount to drive letter
geesefs-win-x64.exe mybucket K:

REM Mount to directory
geesefs-win-x64.exe mybucket C:\mnt\s3

REM With custom endpoint
geesefs-win-x64.exe --endpoint https://s3.amazonaws.com mybucket K:
```

--------------------------------

### Configure S3 Storage Class

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Set the S3 storage class for uploaded objects. Supports STANDARD, STANDARD_IA, and REDUCED_REDUNDANCY. Can also set a minimum size for cold storage.

```bash
# Use Standard storage (default)
geesefs --storage-class STANDARD mybucket /mnt/s3

# Use Infrequent Access
geesefs --storage-class STANDARD_IA mybucket /mnt/s3

# Use Reduced Redundancy
geesefs --storage-class REDUCED_REDUNDANCY mybucket /mnt/s3

# Set minimum size for cold storage (smaller files use STANDARD)
geesefs --storage-class STANDARD_IA --cold-min-size 131072 mybucket /mnt/s3
```

--------------------------------

### GeeseFS Mount with IAM Instance Metadata Authentication

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Illustrates mounting an S3 bucket with GeeseFS using IAM role-based authentication via instance metadata services (Yandex Cloud, AWS IMDSv1, GCP). Supports custom metadata URLs and headers.

```bash
# Use IAM authentication with default GCP flavor
geesefs --iam mybucket /mnt/s3

# Use IAM with AWS IMDSv1 flavor
geesefs --iam --iam-flavor imdsv1 mybucket /mnt/s3

# Use IAM with custom metadata URL
geesefs --iam --iam-url http://custom-metadata-server/token mybucket /mnt/s3

# Use IAM with custom header (Yandex Cloud)
geesefs --iam --iam-header "X-YaCloud-SubjectToken" mybucket /mnt/s3
```

--------------------------------

### Configure S3 Listing API Version

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Set the S3 listing API version for compatibility with different S3 implementations. Supports Yandex extended listing, ListObjectsV2, and ListObjectsV1.

```bash
# Yandex extended listing (default for Yandex)
geesefs --list-type ext-v1 mybucket /mnt/s3

# ListObjectsV2 API (recommended for Amazon S3)
geesefs --list-type 2 mybucket /mnt/s3

# ListObjectsV1 API (legacy)
geesefs --list-type 1 mybucket /mnt/s3
```

--------------------------------

### GeeseFS Memory and Cache Configuration Options

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Details GeeseFS configuration for memory limits and caching behavior, including setting memory limits, metadata entry limits, enabling disk caching, and configuring cache TTL.

```bash
# Set memory limit to 4GB (default is 1GB)
geesefs --memory-limit 4000 mybucket /mnt/s3

# Configure entry limit for metadata cache (default 100000)
geesefs --entry-limit 200000 mybucket /mnt/s3

# Enable disk cache
geesefs --cache /var/cache/geesefs mybucket /mnt/s3

# Configure cache TTL (default 60 seconds)
geesefs --stat-cache-ttl 2m mybucket /mnt/s3

# Return ENOMEM errors when memory limit exceeded
geesefs --use-enomem --memory-limit 2000 mybucket /mnt/s3
```

--------------------------------

### Configure SSE-C Encryption

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Enable Server-Side Encryption with Customer-Provided Keys (SSE-C) for data stored in S3. Requires a base64-encoded key.

```bash
geesefs --sse-c "base64encodedkey==" mybucket /mnt/s3
```

--------------------------------

### Ensure Data Persistence with fsync

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Use the `fsync` system call to ensure data is persisted to S3. Supports syncing individual files or entire directories.

```python
import os
import sys

# Write data to file
with open('/mnt/s3/myfile.txt', 'w') as f:
    f.write('Hello, World!')

# Sync specific file
fd = os.open('/mnt/s3/myfile.txt', os.O_RDONLY)
os.fsync(fd)
os.close(fd)

# Sync entire directory (flushes all changes within)
dir_fd = os.open('/mnt/s3/mydir', os.O_RDONLY)
os.fsync(dir_fd)
os.close(dir_fd)
```

--------------------------------

### GeeseFS Server-Side Encryption Options

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Demonstrates how to enable server-side encryption (SSE-S3 with AES-256) when mounting S3 buckets with GeeseFS.

```bash
# Enable SSE-S3 (AES-256)
geesefs --sse mybucket /mnt/s3
```

--------------------------------

### Set Canned ACL for Uploaded Objects

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure the canned Access Control List (ACL) for uploaded objects. Supports private, public-read, and bucket-owner-full-control.

```bash
# Private (default)
geesefs --acl private mybucket /mnt/s3

# Public read
geesefs --acl public-read mybucket /mnt/s3

# Bucket owner full control
geesefs --acl bucket-owner-full-control mybucket /mnt/s3
```

--------------------------------

### Enable PATCH Method for Partial Updates

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure geesefs to use the PATCH method for efficient partial object updates on Yandex Object Storage. Options include enabling patch, dropping conflicts, and preferring patch for new uploads.

```bash
# Enable PATCH uploads
geesefs --enable-patch mybucket /mnt/s3

# Enable PATCH with conflict dropping
geesefs --enable-patch --drop-patch-conflicts mybucket /mnt/s3

# Prefer PATCH for new uploads (faster visibility, slower upload)
geesefs --enable-patch --prefer-patch-uploads mybucket /mnt/s3
```

--------------------------------

### Mount Azure Blob Storage with Full WASB URI or --endpoint (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README-azure.md

Mounts Azure Blob Storage using GeeseFS with explicit WASB URIs or by specifying the endpoint with the --endpoint flag. This provides flexibility in defining the storage account and container, especially when not relying on default configurations or environment variables. The prefix separator defaults to ':' if not otherwise specified.

```Shell
$ $GOPATH/bin/geesefs wasb://container@myaccount.blob.core.windows.net/prefix <mountpoint>
$ $GOPATH/bin/geesefs --endpoint https://myaccount.blob.core.windows.net wasb://container:prefix <mountpoint>
```

--------------------------------

### Tune Readahead Behavior

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure readahead behavior for different workloads to optimize performance. Options include default settings, optimization for random small reads, large sequential reads, and parallel readahead chunk size.

```bash
# Default readahead (5MB, 100MB large)
geesefs mybucket /mnt/s3

# Optimize for random small reads
geesefs --read-ahead 128 --read-ahead-small 64 --small-read-cutoff 64 mybucket /mnt/s3

# Optimize for large sequential reads
geesefs --read-ahead-large 200000 --large-read-cutoff 10000 mybucket /mnt/s3

# Configure parallel readahead chunk size
geesefs --read-ahead-parallel 40000 mybucket /mnt/s3
```

--------------------------------

### Enable Cluster Mode for Synchronized Caches

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure geesefs to run in cluster mode, allowing synchronized caches across multiple hosts. Requires specifying the current node's ID and address, along with peer addresses.

```bash
# Node 1
geesefs --cluster --cluster-me 1:node1.example.com:5000 \
    --cluster-peer 1:node1.example.com:5000 \
    --cluster-peer 2:node2.example.com:5000 \
    mybucket /mnt/s3

# Node 2
geesefs --cluster --cluster-me 2:node2.example.com:5000 \
    --cluster-peer 1:node1.example.com:5000 \
    --cluster-peer 2:node2.example.com:5000 \
    mybucket /mnt/s3
```

--------------------------------

### Mount Azure Blob Storage with ~/.azure/config (Shell)

Source: https://github.com/yandex-cloud/geesefs/blob/master/README-azure.md

Mounts Azure Blob Storage using GeeseFS by configuring credentials in the ~/.azure/config file. This allows for persistent configuration without setting environment variables for each mount. It supports mounting the entire container or a specific prefix within the container.

```Shell
$ cat ~/.azure/config
[storage]
account = "myblobstorage"
key = "MY-STORAGE-KEY"
$ $GOPATH/bin/geesefs wasb://container <mountpoint>
$ $GOPATH/bin/geesefs wasb://container:prefix <mountpoint> # if you only want to mount objects under a prefix
```

--------------------------------

### GeeseFS Performance Optimization Flags

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Utilize various flags to optimize GeeseFS performance for different use cases. These flags can reduce S3 costs, improve read/write speeds, and adjust filesystem behavior.

```bash
# Reduce S3 operation costs (fewer requests)
geesefs --cheap mybucket /mnt/s3
```

```bash
# Disable content checksums for performance
geesefs --no-checksum mybucket /mnt/s3
```

```bash
# Disable xattrs for faster listings
geesefs --disable-xattr mybucket /mnt/s3
```

```bash
# Disable directory object creation
geesefs --no-dir-object mybucket /mnt/s3
```

```bash
# Assume all directories exist
geesefs --no-implicit-dir mybucket /mnt/s3
```

```bash
# Wait for writes on close (close-to-open consistency)
geesefs --fsync-on-close mybucket /mnt/s3
```

--------------------------------

### Configure SSE-KMS Encryption

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Enable Server-Side Encryption with AWS KMS (SSE-KMS) for data stored in S3. Supports default CMK or a specific KMS key ID.

```bash
geesefs --sse-kms "" mybucket /mnt/s3
geesefs --sse-kms "arn:aws:kms:us-east-1:123456789:key/12345678-1234-1234-1234-123456789012" mybucket /mnt/s3
```

--------------------------------

### Mount Azure Blob Storage Containers

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Mount Azure Blob Storage containers as a filesystem using geesefs. Supports both `wasb://` and `abfs://` URL formats for different Azure storage types.

```bash
# Using wasb:// URL format
geesefs wasb://container@storageaccount.blob.core.windows.net /mnt/azure

# Using abfs:// URL format (Azure Data Lake Storage Gen2)
geesefs abfs://container@storageaccount.dfs.core.windows.net /mnt/azure
```

--------------------------------

### Configure Retry Behavior for GeeseFS

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Configure retry intervals and attempt counts for GeeseFS operations. This helps manage network instability and transient S3 errors by retrying failed requests.

```bash
geesefs --retry-interval 30s --read-retry-attempts 10 mybucket /mnt/s3
```

```bash
geesefs --sdk-max-retries 5 \
    --sdk-min-retry-delay 50ms \
    --sdk-max-retry-delay 5m \
    mybucket /mnt/s3
```

--------------------------------

### Invalidate Cache using xattr or Special Filename

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Force refresh of cached file or directory metadata. Uses `setfattr` on Linux/macOS or a special filename on Windows.

```bash
# Invalidate cache for specific file using xattr
setfattr -n .invalidate /mnt/s3/path/to/file

# Invalidate directory cache
setfattr -n .invalidate /mnt/s3/path/to/directory

# On Windows, access special filename to invalidate
dir /mnt/s3/path/to/directory/.invalidate
```

--------------------------------

### Configure HTTP Timeout and Retry Behavior

Source: https://context7.com/yandex-cloud/geesefs/llms.txt

Set the HTTP timeout for S3 operations. This option controls how long geesefs will wait for a response from the S3 server.

```bash
# Set HTTP timeout
geesefs --http-timeout 60s mybucket /mnt/s3
```

--------------------------------

### Force GeeseFS Cache Invalidation

Source: https://github.com/yandex-cloud/geesefs/blob/master/README.md

This command forcibly refreshes the file or directory cache in GeeseFS. It's used to ensure that the latest file/directory state is read from the server, which is crucial for handling concurrent updates and avoiding stale data.

```bash
setfattr -n .invalidate <filename>
```
Tokens:
Raw