Akave (akave-ai/akave-docs)

Akave

https://github.com/akave-ai/akave-docs
Admin
Akave is a decentralized protocol that empowers individuals and businesses with robust storage...

Tokens:59,150
Snippets:369
Trust Score:7.5
Update:2 weeks ago
Show doc for...
Context Summary (auto-generated)
Raw
# Akave

Akave is a decentralized protocol that provides robust storage solutions and advanced data management tools for on-chain data lakes. The platform combines decentralized infrastructure with blockchain-backed verification, enabling users to gain full control of their data assets while ensuring data privacy, security, and verifiability. Akave offers both a native CLI (`akavecli`) for direct blockchain-integrated storage operations and an S3-compatible API (`Akave O3`) that allows seamless integration with existing AWS tooling.

The platform addresses critical challenges in data management including security vulnerabilities, privacy concerns, and unpredictable costs associated with centralized cloud storage. Akave leverages erasure coding with Reed-Solomon algorithm for data resilience, Proof-of-Data-Possession (PDP) for archival storage verification, and on-chain metadata management for tamper-proof access control. Users can self-host O3 within their own infrastructure or use Akave.Cloud for a managed decentralized storage experience.

## AWS CLI Profile Setup

Configure a dedicated AWS CLI profile for interacting with Akave O3 to streamline authentication and avoid setting environment variables for each session.

```bash
# Configure the AWS CLI profile for Akave O3
aws configure --profile akave-o3

# Enter when prompted:
# AWS Access Key ID: <your-access-key>
# AWS Secret Access Key: <your-secret-key>
# Default region name: akave-network
# Default output format: json

# Or edit credentials file directly
nano ~/.aws/credentials

# Add the following configuration:
# [akave-o3]
# aws_access_key_id = <your_access_key>
# aws_secret_access_key = <your_secret_key>
# endpoint_url = https://o3-rc2.akave.xyz

# Test the configuration
aws s3api list-buckets \
  --profile akave-o3 \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Create Bucket

Create a new bucket in Akave O3 to store objects using either `aws s3api` for granular control or `aws s3` for simplified commands.

```bash
# Using aws s3api
aws s3api create-bucket \
  --bucket my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 mb s3://my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz
```

## List Buckets

List all buckets associated with your account to view available storage containers.

```bash
# Using aws s3api
aws s3api list-buckets \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 ls \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Delete Bucket

Remove an empty bucket from your account. Note that buckets must be empty before deletion, and versioning is enabled by default.

```bash
# Using aws s3api
aws s3api delete-bucket \
  --bucket my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 rb s3://my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Upload Object

Upload a file to an Akave O3 bucket using standard S3 API commands.

```bash
# Using aws s3api
aws s3api put-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  --body ./myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 cp ./myfile.txt s3://my-akave-bucket/myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Download Object

Download a file from an Akave O3 bucket to your local filesystem.

```bash
# Using aws s3api
aws s3api get-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  ./downloaded-myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 cp s3://my-akave-bucket/myfile.txt ./downloaded-myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz
```

## List Objects

List all objects within a bucket and optionally get total size with recursive listing.

```bash
# Using aws s3api
aws s3api list-objects \
  --bucket my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 ls s3://my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Get total size for all objects in a bucket
aws s3 ls s3://my-akave-bucket \
  --recursive --human-readable --summarize \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Delete Object

Remove an object from a bucket. For versioned objects, you must delete each version individually.

```bash
# Using aws s3api
aws s3api delete-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Using aws s3 (simplified)
aws s3 rm s3://my-akave-bucket/myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz
```

## View Object Metadata and eCID

Retrieve object metadata including the encrypted content identifier (eCID) which provides blockchain verification for stored data.

```bash
aws s3api head-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  --endpoint-url=https://o3-rc2.akave.xyz

# Response includes Network-Root-Cid in Metadata:
# {
#     "LastModified": "2024-05-15T00:00:00+00:00",
#     "ContentLength": 2194339,
#     "Metadata": {
#         "Network-Root-Cid": "bafybeicun4bwqcby46mcxicctyob6vjd4lid74k3c4lzfaifl56sghrt3q",
#         "Network-State": "done"
#     }
# }
```

## Object Versioning

Akave O3 has versioning enabled by default for immutable blockchain storage. List, download, and delete specific object versions.

```bash
# List all object versions
aws s3api list-object-versions \
  --bucket my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Download a specific version
aws s3api get-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  --version-id <version-id> \
  ./myfile.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Delete a specific version
aws s3api delete-object \
  --bucket my-akave-bucket \
  --key myfile.txt \
  --version-id <version-id> \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Copy Objects

Copy objects within the same bucket or across different buckets.

```bash
# Copy within the same bucket (using s3api)
aws s3api copy-object \
  --bucket my-akave-bucket \
  --copy-source my-akave-bucket/myfile.txt \
  --key myfile-copy.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Copy using aws s3
aws s3 cp s3://my-akave-bucket/myfile.txt s3://my-akave-bucket/myfile-copy.txt \
  --endpoint-url https://o3-rc2.akave.xyz

# Copy across buckets
aws s3 cp s3://source-bucket/myfile.txt s3://destination-bucket/myfile-copy.txt \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Multipart Upload (Simple)

For large files, the AWS CLI automatically handles multipart uploads when the file exceeds 8 MB.

```bash
# Automatic multipart upload for large files
aws s3 cp ./largefile.zip s3://my-akave-bucket/largefile.zip \
  --endpoint-url https://o3-rc2.akave.xyz

# Control part size for better performance
aws s3 cp ./largefile.zip s3://my-akave-bucket/largefile.zip \
  --part-size 64MB \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Multipart Upload (Granular Control)

Manually manage multipart uploads for fine-grained control over large file transfers.

```bash
# Step 1: Initiate multipart upload (returns UploadId)
aws s3api create-multipart-upload \
  --bucket my-akave-bucket \
  --key largefile.zip \
  --endpoint-url https://o3-rc2.akave.xyz

# Step 2: Upload each part (repeat for each part, incrementing part-number)
aws s3api upload-part \
  --bucket my-akave-bucket \
  --key largefile.zip \
  --part-number 1 \
  --body part1.zip \
  --upload-id <UploadId> \
  --endpoint-url https://o3-rc2.akave.xyz

# Step 3: Complete multipart upload with parts.json
aws s3api complete-multipart-upload \
  --bucket my-akave-bucket \
  --key largefile.zip \
  --upload-id <UploadId> \
  --multipart-upload file://parts.json \
  --endpoint-url https://o3-rc2.akave.xyz

# parts.json format:
# {
#   "Parts": [
#     { "ETag": "\"etag-part1\"", "PartNumber": 1 },
#     { "ETag": "\"etag-part2\"", "PartNumber": 2 }
#   ]
# }

# Abort multipart upload if needed
aws s3api abort-multipart-upload \
  --bucket my-akave-bucket \
  --key largefile.zip \
  --upload-id <UploadId> \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Generate Presigned URLs

Create time-limited, signed URLs for secure file sharing without exposing credentials.

```bash
# Generate a presigned URL for download (valid for 1 hour)
aws s3 presign s3://my-akave-bucket/myfile.txt \
  --expires-in 3600 \
  --endpoint-url https://o3-rc2.akave.xyz

# Use the presigned URL to download
curl "https://o3-rc2.akave.xyz/my-akave-bucket/myfile.txt?...signature..."
```

## Server-Side Encryption

Enable server-side encryption at the bucket level for automatic encryption of all uploaded objects.

```bash
# Enable SSE by default on a bucket
aws s3api put-bucket-encryption \
  --bucket my-akave-bucket \
  --server-side-encryption-configuration file://sse.json \
  --endpoint-url https://o3-rc2.akave.xyz

# sse.json content:
# {
#   "Rules": [
#     {
#       "ApplyServerSideEncryptionByDefault": {
#         "SSEAlgorithm": "AES256"
#       }
#     }
#   ]
# }

# View bucket encryption setting
aws s3api get-bucket-encryption \
  --bucket my-akave-bucket \
  --endpoint-url https://o3-rc2.akave.xyz

# Upload with explicit encryption
aws s3api put-object \
  --bucket my-akave-bucket \
  --key encrypted.txt \
  --body myfile.txt \
  --server-side-encryption AES256 \
  --endpoint-url https://o3-rc2.akave.xyz
```

## Akave CLI Installation

Install and build the Akave CLI (`akavecli`) for direct blockchain-integrated storage operations.

```bash
# Clone the repository
git clone https://github.com/akave-ai/akavesdk.git
cd akavesdk

# Build the CLI
make build

# Move binary to system PATH
sudo mv bin/akavecli /usr/local/bin/

# Verify installation
akavecli version
```

## Akave CLI Wallet Management

Manage wallets for blockchain-integrated storage operations using the built-in wallet subsystem.

```bash
# Create a new wallet
akavecli wallet create

# List all wallets
akavecli wallet list

# Import a private key
akavecli wallet import <wallet-name> <private-key>

# Export a private key
akavecli wallet export-key <wallet-name>

# Check wallet balance
akavecli wallet balance <wallet-name>

# Use wallet in commands
akavecli bucket create mybucket \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500
```

## Akave CLI Bucket Operations

Create, list, view, and delete buckets using the Akave CLI with blockchain integration.

```bash
# Create a bucket
akavecli bucket create <bucket-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# List all buckets
akavecli bucket list \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# View bucket details
akavecli bucket view <bucket-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# Delete a bucket
akavecli bucket delete <bucket-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500
```

## Akave CLI File Operations

Upload, download, list, and delete files using the Akave CLI with optional encryption.

```bash
# Upload a file (max 5 GB per upload)
akavecli file upload <bucket-name> <file-path> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# Upload with encryption
akavecli file upload <bucket-name> <file-path> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500 \
  --metadata-encryption \
  --encryption-key "<key>"

# List files in a bucket
akavecli file list <bucket-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# Get file info
akavecli file info <bucket-name> <file-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# Download a file
akavecli file download <bucket-name> <file-name> <destination-folder> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500

# Delete a file
akavecli file delete <bucket-name> <file-name> \
  --account <wallet-name> \
  --node-address connect.akave.ai:5500
```

## PDP Archival Storage

Use Proof-of-Data-Possession for long-term archival storage backed by Filecoin storage providers.

```bash
# Upload data for archival (use port 9500 for archival tier)
akavecli file upload <bucket> <file> \
  --account <wallet-name> \
  --node-address connect.akave.ai:9500

# Check archival status
akavecli archival-metadata <bucket> <file> \
  --account <wallet-name> \
  --node-address connect.akave.ai:9500

# Check with verbose output
akavecli archival-metadata <bucket> <file> \
  --account <wallet-name> \
  --node-address connect.akave.ai:9500 \
  --verbose

# Download from archival storage
akavecli file download <bucket> <file> <output-dir> \
  --archival \
  --account <wallet-name> \
  --node-address connect.akave.ai:9500
```

## Rclone Integration

Use Rclone to migrate data to and from Akave with S3-compatible configuration.

```bash
# Configure Rclone for Akave
rclone config
# Select: n (new remote)
# Name: Akave
# Storage: Amazon S3 Compliant Storage Providers
# Provider: Any other S3 compatible provider
# Access Key ID: <your-access-key>
# Secret Access Key: <your-secret-key>
# Region: akave-network
# Endpoint: https://o3-rc2.akave.xyz

# Create bucket
rclone mkdir Akave:<bucket-name>

# List buckets
rclone lsd Akave:

# Upload file
rclone copy <file-path> Akave:<bucket-name>

# Download file
rclone copyto Akave:<bucket-name>/<file-path> <local-path>

# Migrate data from AWS S3 to Akave
rclone sync s3:<bucket-name> Akave:<bucket-name> --progress

# Validate migration
rclone check s3:<bucket-name> Akave:<bucket-name> --size-only
```

## S3FS Mount

Mount Akave buckets as a local filesystem using s3fs-fuse for native file operations.

```bash
# Install s3fs (macOS)
brew install --cask macfuse
brew install gromgit/fuse/s3fs-mac

# Install s3fs (Linux)
sudo apt install s3fs

# Create mount directory
mkdir -p ~/akave-mount

# Mount bucket
s3fs your-bucket-name ~/akave-mount \
  -o url=https://o3-rc2.akave.xyz \
  -o profile=akave-o3

# Use standard file operations
ls -l ~/akave-mount
cp myfile.txt ~/akave-mount/
cp ~/akave-mount/myfile.txt ./

# Unmount when done
umount ~/akave-mount
```

## S3FS Python Integration

Use the s3fs Python library for programmatic access to Akave storage with pandas integration.

```python
import s3fs
import os
import pandas as pd

# Initialize S3FileSystem with Akave credentials
fs = s3fs.S3FileSystem(
    key=os.environ.get("AKAVE_ACCESS_KEY"),
    secret=os.environ.get("AKAVE_SECRET_KEY"),
    endpoint_url="https://o3-rc2.akave.xyz",
    client_kwargs={"region_name": "akave-network"}
)

# Or use AWS CLI profile
fs = s3fs.S3FileSystem(
    profile="akave-o3",
    endpoint_url="https://o3-rc2.akave.xyz",
    client_kwargs={"region_name": "akave-network"}
)

# List buckets
buckets = fs.ls("")
print(f"Available buckets: {buckets}")

# List files in a bucket
files = fs.ls("your-bucket-name")
for file in files:
    print(file)

# Upload a file
fs.put("local-file.txt", "your-bucket-name/remote-file.txt")

# Download a file
fs.get("your-bucket-name/remote-file.txt", "downloaded-file.txt")

# Delete a file
fs.rm("your-bucket-name/file-to-delete.txt")

# Read CSV directly from Akave
df = pd.read_csv(fs.open("your-bucket-name/data.csv"))

# Write DataFrame to Akave as parquet
df.to_parquet(fs.open("your-bucket-name/processed-data.parquet", "wb"))

# Get file metadata
info = fs.info("your-bucket-name/myfile.txt")
print(f"File size: {info['size']} bytes")
```

## DuckDB Integration

Query data directly from Akave storage using DuckDB's in-memory SQL engine.

```bash
# Install DuckDB
brew install duckdb  # macOS
# or
curl https://install.duckdb.org | sh  # Linux

# Start DuckDB CLI
duckdb
```

```sql
-- Install and load httpfs extension
INSTALL httpfs;
LOAD httpfs;

-- Configure Akave credentials
CREATE OR REPLACE PERSISTENT SECRET akave_secret (
    TYPE s3,
    PROVIDER config,
    KEY_ID '<your-access-key>',
    SECRET '<your-secret-key>',
    REGION 'akave-network',
    ENDPOINT 'o3-rc2.akave.xyz'
);

-- Query data directly from Akave
SELECT * FROM 's3://bucket/file.parquet';

-- Read multiple parquet files
SELECT * FROM read_parquet([
  's3://bucket/file1.parquet',
  's3://bucket/file2.parquet'
]);

-- Read all parquet files with wildcard
SELECT * FROM read_parquet('s3://bucket/*.parquet');

-- Count rows without loading entire file
SELECT count(*) FROM 's3://bucket/file.parquet';

-- Select specific columns
SELECT column1, column2 FROM 's3://bucket/file.parquet';

-- Attach Akave database
ATTACH 's3://bucket/database.duckdb' AS akave;
SELECT * FROM akave.table_name;
```

## Hugging Face Datasets Integration

Store and retrieve machine learning datasets on Akave using Hugging Face datasets library.

```python
# Install dependencies
# pip install s3fs datasets

from huggingface_s3 import HuggingFaceS3
from datasets import load_dataset

# Initialize the Hugging Face S3 client
hf_s3 = HuggingFaceS3()

# List available buckets
buckets = hf_s3.list_buckets()
print(buckets)

# Transfer a Hugging Face dataset to Akave
output_dir = hf_s3.transfer_dataset("mnist")
output_dir = hf_s3.transfer_dataset("imdb", output_path="text/imdb_dataset")

# Save a processed dataset to Akave
dataset = load_dataset("imdb", split="train")
save_path = hf_s3.save_dataset(dataset, output_path="processed/imdb_train")

# Load a dataset from Akave
dataset = hf_s3.load_dataset(path="imdb")
print(f"Dataset has {len(dataset)} examples")
print(dataset[0])
```

Akave provides a comprehensive decentralized storage solution suitable for enterprises needing data sovereignty, AI/ML pipelines storing training data and model states, backup and archiving applications, and DePIN networks requiring secure, verifiable storage. The S3-compatible API ensures seamless migration from existing cloud infrastructure while blockchain-backed verification provides tamper-proof data integrity and access control.

Integration patterns include using AWS CLI and SDKs for standard S3 operations, mounting Akave buckets as local filesystems with s3fs-fuse, data migration with Rclone, direct analytics queries with DuckDB, and ML dataset management with Hugging Face. For long-term archival, the PDP storage tier leverages Filecoin's decentralized network with cryptographic proofs of data possession. All operations support encryption, versioning, and on-chain access control for enterprise-grade security and compliance requirements.