# Akave Akave is a decentralized protocol that provides robust storage solutions and advanced data management tools for on-chain data lakes. The platform combines decentralized infrastructure with blockchain-backed verification, enabling users to gain full control of their data assets while ensuring data privacy, security, and verifiability. Akave offers both a native CLI (`akavecli`) for direct blockchain-integrated storage operations and an S3-compatible API (`Akave O3`) that allows seamless integration with existing AWS tooling. The platform addresses critical challenges in data management including security vulnerabilities, privacy concerns, and unpredictable costs associated with centralized cloud storage. Akave leverages erasure coding with Reed-Solomon algorithm for data resilience, Proof-of-Data-Possession (PDP) for archival storage verification, and on-chain metadata management for tamper-proof access control. Users can self-host O3 within their own infrastructure or use Akave.Cloud for a managed decentralized storage experience. ## AWS CLI Profile Setup Configure a dedicated AWS CLI profile for interacting with Akave O3 to streamline authentication and avoid setting environment variables for each session. ```bash # Configure the AWS CLI profile for Akave O3 aws configure --profile akave-o3 # Enter when prompted: # AWS Access Key ID: # AWS Secret Access Key: # Default region name: akave-network # Default output format: json # Or edit credentials file directly nano ~/.aws/credentials # Add the following configuration: # [akave-o3] # aws_access_key_id = # aws_secret_access_key = # endpoint_url = https://o3-rc2.akave.xyz # Test the configuration aws s3api list-buckets \ --profile akave-o3 \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Create Bucket Create a new bucket in Akave O3 to store objects using either `aws s3api` for granular control or `aws s3` for simplified commands. ```bash # Using aws s3api aws s3api create-bucket \ --bucket my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 mb s3://my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz ``` ## List Buckets List all buckets associated with your account to view available storage containers. ```bash # Using aws s3api aws s3api list-buckets \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 ls \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Delete Bucket Remove an empty bucket from your account. Note that buckets must be empty before deletion, and versioning is enabled by default. ```bash # Using aws s3api aws s3api delete-bucket \ --bucket my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 rb s3://my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Upload Object Upload a file to an Akave O3 bucket using standard S3 API commands. ```bash # Using aws s3api aws s3api put-object \ --bucket my-akave-bucket \ --key myfile.txt \ --body ./myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 cp ./myfile.txt s3://my-akave-bucket/myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Download Object Download a file from an Akave O3 bucket to your local filesystem. ```bash # Using aws s3api aws s3api get-object \ --bucket my-akave-bucket \ --key myfile.txt \ ./downloaded-myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 cp s3://my-akave-bucket/myfile.txt ./downloaded-myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz ``` ## List Objects List all objects within a bucket and optionally get total size with recursive listing. ```bash # Using aws s3api aws s3api list-objects \ --bucket my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 ls s3://my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Get total size for all objects in a bucket aws s3 ls s3://my-akave-bucket \ --recursive --human-readable --summarize \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Delete Object Remove an object from a bucket. For versioned objects, you must delete each version individually. ```bash # Using aws s3api aws s3api delete-object \ --bucket my-akave-bucket \ --key myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz # Using aws s3 (simplified) aws s3 rm s3://my-akave-bucket/myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz ``` ## View Object Metadata and eCID Retrieve object metadata including the encrypted content identifier (eCID) which provides blockchain verification for stored data. ```bash aws s3api head-object \ --bucket my-akave-bucket \ --key myfile.txt \ --endpoint-url=https://o3-rc2.akave.xyz # Response includes Network-Root-Cid in Metadata: # { # "LastModified": "2024-05-15T00:00:00+00:00", # "ContentLength": 2194339, # "Metadata": { # "Network-Root-Cid": "bafybeicun4bwqcby46mcxicctyob6vjd4lid74k3c4lzfaifl56sghrt3q", # "Network-State": "done" # } # } ``` ## Object Versioning Akave O3 has versioning enabled by default for immutable blockchain storage. List, download, and delete specific object versions. ```bash # List all object versions aws s3api list-object-versions \ --bucket my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Download a specific version aws s3api get-object \ --bucket my-akave-bucket \ --key myfile.txt \ --version-id \ ./myfile.txt \ --endpoint-url https://o3-rc2.akave.xyz # Delete a specific version aws s3api delete-object \ --bucket my-akave-bucket \ --key myfile.txt \ --version-id \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Copy Objects Copy objects within the same bucket or across different buckets. ```bash # Copy within the same bucket (using s3api) aws s3api copy-object \ --bucket my-akave-bucket \ --copy-source my-akave-bucket/myfile.txt \ --key myfile-copy.txt \ --endpoint-url https://o3-rc2.akave.xyz # Copy using aws s3 aws s3 cp s3://my-akave-bucket/myfile.txt s3://my-akave-bucket/myfile-copy.txt \ --endpoint-url https://o3-rc2.akave.xyz # Copy across buckets aws s3 cp s3://source-bucket/myfile.txt s3://destination-bucket/myfile-copy.txt \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Multipart Upload (Simple) For large files, the AWS CLI automatically handles multipart uploads when the file exceeds 8 MB. ```bash # Automatic multipart upload for large files aws s3 cp ./largefile.zip s3://my-akave-bucket/largefile.zip \ --endpoint-url https://o3-rc2.akave.xyz # Control part size for better performance aws s3 cp ./largefile.zip s3://my-akave-bucket/largefile.zip \ --part-size 64MB \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Multipart Upload (Granular Control) Manually manage multipart uploads for fine-grained control over large file transfers. ```bash # Step 1: Initiate multipart upload (returns UploadId) aws s3api create-multipart-upload \ --bucket my-akave-bucket \ --key largefile.zip \ --endpoint-url https://o3-rc2.akave.xyz # Step 2: Upload each part (repeat for each part, incrementing part-number) aws s3api upload-part \ --bucket my-akave-bucket \ --key largefile.zip \ --part-number 1 \ --body part1.zip \ --upload-id \ --endpoint-url https://o3-rc2.akave.xyz # Step 3: Complete multipart upload with parts.json aws s3api complete-multipart-upload \ --bucket my-akave-bucket \ --key largefile.zip \ --upload-id \ --multipart-upload file://parts.json \ --endpoint-url https://o3-rc2.akave.xyz # parts.json format: # { # "Parts": [ # { "ETag": "\"etag-part1\"", "PartNumber": 1 }, # { "ETag": "\"etag-part2\"", "PartNumber": 2 } # ] # } # Abort multipart upload if needed aws s3api abort-multipart-upload \ --bucket my-akave-bucket \ --key largefile.zip \ --upload-id \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Generate Presigned URLs Create time-limited, signed URLs for secure file sharing without exposing credentials. ```bash # Generate a presigned URL for download (valid for 1 hour) aws s3 presign s3://my-akave-bucket/myfile.txt \ --expires-in 3600 \ --endpoint-url https://o3-rc2.akave.xyz # Use the presigned URL to download curl "https://o3-rc2.akave.xyz/my-akave-bucket/myfile.txt?...signature..." ``` ## Server-Side Encryption Enable server-side encryption at the bucket level for automatic encryption of all uploaded objects. ```bash # Enable SSE by default on a bucket aws s3api put-bucket-encryption \ --bucket my-akave-bucket \ --server-side-encryption-configuration file://sse.json \ --endpoint-url https://o3-rc2.akave.xyz # sse.json content: # { # "Rules": [ # { # "ApplyServerSideEncryptionByDefault": { # "SSEAlgorithm": "AES256" # } # } # ] # } # View bucket encryption setting aws s3api get-bucket-encryption \ --bucket my-akave-bucket \ --endpoint-url https://o3-rc2.akave.xyz # Upload with explicit encryption aws s3api put-object \ --bucket my-akave-bucket \ --key encrypted.txt \ --body myfile.txt \ --server-side-encryption AES256 \ --endpoint-url https://o3-rc2.akave.xyz ``` ## Akave CLI Installation Install and build the Akave CLI (`akavecli`) for direct blockchain-integrated storage operations. ```bash # Clone the repository git clone https://github.com/akave-ai/akavesdk.git cd akavesdk # Build the CLI make build # Move binary to system PATH sudo mv bin/akavecli /usr/local/bin/ # Verify installation akavecli version ``` ## Akave CLI Wallet Management Manage wallets for blockchain-integrated storage operations using the built-in wallet subsystem. ```bash # Create a new wallet akavecli wallet create # List all wallets akavecli wallet list # Import a private key akavecli wallet import # Export a private key akavecli wallet export-key # Check wallet balance akavecli wallet balance # Use wallet in commands akavecli bucket create mybucket \ --account \ --node-address connect.akave.ai:5500 ``` ## Akave CLI Bucket Operations Create, list, view, and delete buckets using the Akave CLI with blockchain integration. ```bash # Create a bucket akavecli bucket create \ --account \ --node-address connect.akave.ai:5500 # List all buckets akavecli bucket list \ --account \ --node-address connect.akave.ai:5500 # View bucket details akavecli bucket view \ --account \ --node-address connect.akave.ai:5500 # Delete a bucket akavecli bucket delete \ --account \ --node-address connect.akave.ai:5500 ``` ## Akave CLI File Operations Upload, download, list, and delete files using the Akave CLI with optional encryption. ```bash # Upload a file (max 5 GB per upload) akavecli file upload \ --account \ --node-address connect.akave.ai:5500 # Upload with encryption akavecli file upload \ --account \ --node-address connect.akave.ai:5500 \ --metadata-encryption \ --encryption-key "" # List files in a bucket akavecli file list \ --account \ --node-address connect.akave.ai:5500 # Get file info akavecli file info \ --account \ --node-address connect.akave.ai:5500 # Download a file akavecli file download \ --account \ --node-address connect.akave.ai:5500 # Delete a file akavecli file delete \ --account \ --node-address connect.akave.ai:5500 ``` ## PDP Archival Storage Use Proof-of-Data-Possession for long-term archival storage backed by Filecoin storage providers. ```bash # Upload data for archival (use port 9500 for archival tier) akavecli file upload \ --account \ --node-address connect.akave.ai:9500 # Check archival status akavecli archival-metadata \ --account \ --node-address connect.akave.ai:9500 # Check with verbose output akavecli archival-metadata \ --account \ --node-address connect.akave.ai:9500 \ --verbose # Download from archival storage akavecli file download \ --archival \ --account \ --node-address connect.akave.ai:9500 ``` ## Rclone Integration Use Rclone to migrate data to and from Akave with S3-compatible configuration. ```bash # Configure Rclone for Akave rclone config # Select: n (new remote) # Name: Akave # Storage: Amazon S3 Compliant Storage Providers # Provider: Any other S3 compatible provider # Access Key ID: # Secret Access Key: # Region: akave-network # Endpoint: https://o3-rc2.akave.xyz # Create bucket rclone mkdir Akave: # List buckets rclone lsd Akave: # Upload file rclone copy Akave: # Download file rclone copyto Akave:/ # Migrate data from AWS S3 to Akave rclone sync s3: Akave: --progress # Validate migration rclone check s3: Akave: --size-only ``` ## S3FS Mount Mount Akave buckets as a local filesystem using s3fs-fuse for native file operations. ```bash # Install s3fs (macOS) brew install --cask macfuse brew install gromgit/fuse/s3fs-mac # Install s3fs (Linux) sudo apt install s3fs # Create mount directory mkdir -p ~/akave-mount # Mount bucket s3fs your-bucket-name ~/akave-mount \ -o url=https://o3-rc2.akave.xyz \ -o profile=akave-o3 # Use standard file operations ls -l ~/akave-mount cp myfile.txt ~/akave-mount/ cp ~/akave-mount/myfile.txt ./ # Unmount when done umount ~/akave-mount ``` ## S3FS Python Integration Use the s3fs Python library for programmatic access to Akave storage with pandas integration. ```python import s3fs import os import pandas as pd # Initialize S3FileSystem with Akave credentials fs = s3fs.S3FileSystem( key=os.environ.get("AKAVE_ACCESS_KEY"), secret=os.environ.get("AKAVE_SECRET_KEY"), endpoint_url="https://o3-rc2.akave.xyz", client_kwargs={"region_name": "akave-network"} ) # Or use AWS CLI profile fs = s3fs.S3FileSystem( profile="akave-o3", endpoint_url="https://o3-rc2.akave.xyz", client_kwargs={"region_name": "akave-network"} ) # List buckets buckets = fs.ls("") print(f"Available buckets: {buckets}") # List files in a bucket files = fs.ls("your-bucket-name") for file in files: print(file) # Upload a file fs.put("local-file.txt", "your-bucket-name/remote-file.txt") # Download a file fs.get("your-bucket-name/remote-file.txt", "downloaded-file.txt") # Delete a file fs.rm("your-bucket-name/file-to-delete.txt") # Read CSV directly from Akave df = pd.read_csv(fs.open("your-bucket-name/data.csv")) # Write DataFrame to Akave as parquet df.to_parquet(fs.open("your-bucket-name/processed-data.parquet", "wb")) # Get file metadata info = fs.info("your-bucket-name/myfile.txt") print(f"File size: {info['size']} bytes") ``` ## DuckDB Integration Query data directly from Akave storage using DuckDB's in-memory SQL engine. ```bash # Install DuckDB brew install duckdb # macOS # or curl https://install.duckdb.org | sh # Linux # Start DuckDB CLI duckdb ``` ```sql -- Install and load httpfs extension INSTALL httpfs; LOAD httpfs; -- Configure Akave credentials CREATE OR REPLACE PERSISTENT SECRET akave_secret ( TYPE s3, PROVIDER config, KEY_ID '', SECRET '', REGION 'akave-network', ENDPOINT 'o3-rc2.akave.xyz' ); -- Query data directly from Akave SELECT * FROM 's3://bucket/file.parquet'; -- Read multiple parquet files SELECT * FROM read_parquet([ 's3://bucket/file1.parquet', 's3://bucket/file2.parquet' ]); -- Read all parquet files with wildcard SELECT * FROM read_parquet('s3://bucket/*.parquet'); -- Count rows without loading entire file SELECT count(*) FROM 's3://bucket/file.parquet'; -- Select specific columns SELECT column1, column2 FROM 's3://bucket/file.parquet'; -- Attach Akave database ATTACH 's3://bucket/database.duckdb' AS akave; SELECT * FROM akave.table_name; ``` ## Hugging Face Datasets Integration Store and retrieve machine learning datasets on Akave using Hugging Face datasets library. ```python # Install dependencies # pip install s3fs datasets from huggingface_s3 import HuggingFaceS3 from datasets import load_dataset # Initialize the Hugging Face S3 client hf_s3 = HuggingFaceS3() # List available buckets buckets = hf_s3.list_buckets() print(buckets) # Transfer a Hugging Face dataset to Akave output_dir = hf_s3.transfer_dataset("mnist") output_dir = hf_s3.transfer_dataset("imdb", output_path="text/imdb_dataset") # Save a processed dataset to Akave dataset = load_dataset("imdb", split="train") save_path = hf_s3.save_dataset(dataset, output_path="processed/imdb_train") # Load a dataset from Akave dataset = hf_s3.load_dataset(path="imdb") print(f"Dataset has {len(dataset)} examples") print(dataset[0]) ``` Akave provides a comprehensive decentralized storage solution suitable for enterprises needing data sovereignty, AI/ML pipelines storing training data and model states, backup and archiving applications, and DePIN networks requiring secure, verifiable storage. The S3-compatible API ensures seamless migration from existing cloud infrastructure while blockchain-backed verification provides tamper-proof data integrity and access control. Integration patterns include using AWS CLI and SDKs for standard S3 operations, mounting Akave buckets as local filesystems with s3fs-fuse, data migration with Rclone, direct analytics queries with DuckDB, and ML dataset management with Hugging Face. For long-term archival, the PDP storage tier leverages Filecoin's decentralized network with cryptographic proofs of data possession. All operations support encryption, versioning, and on-chain access control for enterprise-grade security and compliance requirements.