Storj

https://storj.dev/dcs/
Admin
Storj is an enterprise-grade, globally distributed, S3-compatible cloud object storage platform...
Tokens:110,748
Snippets:659
Trust Score:7.6
Update:11 months ago
Show doc for...
Tokens:
Raw
### Install Storj Uplink CLI (Linux AMD64)

Source: https://storj.dev/dcs/third-party-tools/livepeer

Installs the Storj uplink command-line interface on Linux AMD64 systems. This tool allows direct communication with Storage Nodes. It downloads the latest version, unzips it, and installs it to the system's PATH.

```bash
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip
unzip -o uplink_linux_amd64.zip
sudo install uplink /usr/local/bin/uplink
```

--------------------------------

### Execute transcoding script - Shell

Source: https://storj.dev/dcs/third-party-tools/livepeer

Commands to make the script executable and run it. This starts the transcoding process with the specified parameters.

```shell
chmod +x livepeer-req
./livepeer-req
```

--------------------------------

### Clone Rucio and Start Docker Container

Source: https://storj.dev/dcs/third-party-tools/rucio

This snippet clones the Rucio repository, navigates into the directory, starts the Rucio development environment using Docker Compose, and then enters the Rucio Docker container. Ensure Docker is installed and running prior to execution. The container name might vary, use `docker ps` to find the correct name if `dev_rucio_1` is not found.

```bash
git clone https://github.com/rucio/rucio

cd rucio

docker-compose --file ./etc/docker/dev/docker-compose.yml up -d

docker exec -it dev_rucio_1 /bin/bash
```

--------------------------------

### Install MongoDB Total Cluster using Git Clone and Helm

Source: https://storj.dev/dcs/third-party-tools/mongodb

This sequence clones the total-cluster repository from GitHub and installs the MongoDB chart using Helm. It deploys the full MongoDB data platform including Ops Manager, requiring 5-10 minutes for completion due to multiple components. Dependencies include Helm and a running Kubernetes cluster with S3 Gateway; outputs confirm the installation process.

```bash
git clone https://github.com/jasonmimick/total-cluster

helm install mongodb total-cluster
```

--------------------------------

### Install ActiveStorage in Rails project

Source: https://storj.dev/dcs/code/rails-activestorage

Executes the ActiveStorage installation generator to create necessary database migrations and configuration files. Sets up the database schema for file attachments using ActiveStorage. Must be run from within the Rails application root directory.

```shell
rails active_storage:install
```

--------------------------------

### Install Python and Boto3 for Storj SDK

Source: https://storj.dev/dcs/api/s3/presigned-urls

Installs necessary Python packages including boto3 and requests, which are required for interacting with the Storj S3-compatible gateway. This is a prerequisite for running the Python scripts.

```bash
# Install python

brew install python3

# install boto3

pip3 install boto3

pip3 install requests

```

--------------------------------

### Install rclone on Linux/macOS

Source: https://storj.dev/dcs/getting-started

Installs rclone using the official installation script. Requires curl and sudo privileges. This is the primary method for setting up rclone with Storj.

```bash
sudo -v ; curl https://rclone.org/install.sh | sudo bash
```

--------------------------------

### Install uplink CLI for Linux AMD64

Source: https://storj.dev/dcs/third-party-tools/chainstate-snapshots

Installs the Storj uplink CLI on Linux AMD64 systems. It downloads the latest release, unzips it, and moves the binary to the system's PATH. Ensure you have `curl` and `unzip` installed.

```shell
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip

unzip -o uplink_linux_amd64.zip

sudo install uplink /usr/local/bin/uplink
```

--------------------------------

### Setup Storj Credentials in Python

Source: https://storj.dev/dcs/third-party-tools/hugging-face

This Python code securely prompts for Storj access key and secret key, then initializes an S3FileSystem object with the provided endpoint. It uses the getpass library to obfuscate input and requires s3fs installed. The resulting fs object enables interaction with Storj via S3-compatible API.

```python
from getpass import getpass

key = getpass('Enter Storj access key')

secret = getpass('Enter Storj secret key')

import s3fs

storage_options={"key":key, "secret":secret, "client_kwargs": {'endpoint_url':"https://gateway.storjshare.io"}}

fs = s3fs.S3FileSystem(**storage_options)
```

--------------------------------

### Install Uplink CLI on Linux using curl and unzip

Source: https://storj.dev/dcs/api/uplink-cli/installation

Downloads and installs the Storj Uplink CLI tool on Linux systems from official GitHub releases. Uses curl to fetch architecture-specific archives, unzip to extract the binary, and sudo to install it system-wide in /usr/local/bin. Requires curl, unzip, internet connectivity, and sudo privileges.

```shell
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip

unzip -o uplink_linux_amd64.zip

sudo install uplink /usr/local/bin/uplink
```

```shell
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_arm.zip -o uplink_linux_arm.zip

unzip -o uplink_linux_arm.zip

sudo install uplink /usr/local/bin/uplink
```

```shell
curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_arm64.zip -o uplink_linux_arm64.zip

unzip -o uplink_linux_arm64.zip

sudo install uplink /usr/local/bin/uplink
```

--------------------------------

### Initialize Active Storage in JavaScript controller

Source: https://storj.dev/dcs/code/rails-activestorage

Sets up Active Storage client-side functionality by importing the module and starting the service. Integrates with Stimulus application controller pattern.

```javascript
import * as ActiveStorage from '@rails/activestorage'

const application = Application.start()
ActiveStorage.start()
```

--------------------------------

### Install Dependencies with pip

Source: https://storj.dev/dcs/third-party-tools/hugging-face

This shell command installs the necessary Python packages s3fs and datasets using pip for quiet updating. It requires pip to be installed and network access to PyPI. Run this in a terminal or Colab environment before executing Python code.

```shell
pip install -qqU s3fs datasets
```

--------------------------------

### Install Cyberduck on Linux (DEB) using APT

Source: https://storj.dev/dcs/third-party-tools/cyberduck

Installs Cyberduck on Linux distributions that use the APT package manager (like Ubuntu). This involves adding the Cyberduck repository, importing the GPG key, updating the package list, and then installing the 'duck' package.

```bash
echo -e "deb https://s3.amazonaws.com/repo.deb.cyberduck.io stable main" | sudo tee /etc/apt/sources.list.d/cyberduck.list > /dev/null

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys FE7097963FEFBE72

sudo apt-get update

sudo apt-get install duck
```

--------------------------------

### Initialize LucidLink Filespace via Command Line

Source: https://storj.dev/dcs/third-party-tools/lucidlink

Command-line initialization of a LucidLink Filespace with Storj backend. Requires access key, secret key, bucket name, and other configuration parameters. Ideal for automated setups or advanced configurations.

```bash
lucid init-s3 --fs <filespace.domain> --password <rootpassword> --https --endpoint storage.googleapis.com --region <region> --access-key <accesskey> --secret-key <secretkey> --bucket-name <bucket> --provider <provider>
```

```bash
lucid init-s3 --fs <storj.storjdemo> --password <MtrU76#pxxY98> --https --endpoint us1.storj.io/ --region <US east> --access-key <jx626ak4gis37hyvfi42znkz5mrq> --secret-key <j2gyivv7jor6gasbgqkhsalx4jky2qj36zlamm76mxiggl6esaj2u> --bucket-name <lucidlink-bucket> --provider Storj
```

--------------------------------

### Set up Uplink CLI

Source: https://storj.dev/dcs/access

This command initializes the Uplink CLI on your system. It's a prerequisite for using Storj's command-line tools to interact with your storage. Ensure you have the CLI installed before running this.

```bash
uplink setup
```

--------------------------------

### Share static site via Uplink CLI with DNS and TLS

Source: https://storj.dev/dcs/code/static-site-hosting

Demonstrates using the Uplink CLI to share a bucket/prefix, enable DNS linking, and optionally activate TLS. Requires an access grant and replaces placeholders with your hostname, bucket, and prefix. The command outputs a DNS zone file for further configuration.

```shell
uplink share --dns <hostname> sj://<bucket>/<prefix> --tls --not-after=none
```

```shell
uplink share --dns www.example.com sj://bucket/prefix
```

--------------------------------

### Add AWS SDK S3 gem to Gemfile

Source: https://storj.dev/dcs/code/rails-activestorage

Adds the aws-sdk-s3 gem to enable S3-compatible storage support in Rails ActiveStorage. The require: false option prevents autoloading until ActiveStorage explicitly requires it. After adding, run bundle install to install the dependency.

```ruby
gem "aws-sdk-s3", require: false
```

--------------------------------

### Create a bucket with rclone

Source: https://storj.dev/dcs/getting-started

Creates a new bucket in Storj named 'my-bucket'. Requires prior configuration of rclone with Storj credentials.

```bash
$ rclone mkdir storj:my-bucket
```

--------------------------------

### Download Video File

Source: https://storj.dev/dcs/third-party-tools/livepeer

Downloads a sample video file from a provided URL using wget. This file will be uploaded to the Storj bucket as a source file for transcoding.

```bash
wget https://link.storjshare.io/raw/jxztgqgoyipublszkwo4e2gg5n4a/livepeer/global-upload.mp4
```

--------------------------------

### Install Storj S3 Gateway using Helm

Source: https://storj.dev/dcs/third-party-tools/mongodb

This command installs the Tardigrade S3 Gateway chart in the total-cluster namespace using Helm version 3. It requires a pre-configured Kubernetes cluster and Storj access credentials. The gateway enables S3-compatible storage for MongoDB backups on Storj. Outputs confirm successful deployment with pod, service, deployment, and replica set status.

```bash
helm install s3-gateway tardigrade-gateway

NAME: s3-gateway

LAST DEPLOYED: Sat May 30 07:52:43 2020

NAMESPACE: total-cluster

STATUS: deployed

REVISION: 1

TEST SUITE: None
```

--------------------------------

### Upload File using Presigned URL with Curl

Source: https://storj.dev/dcs/api/s3/presigned-urls

Example `curl` command to upload a file to Storj using a generated presigned URL. This demonstrates how to perform the actual upload operation after obtaining the URL.

```bash
curl -v --upload-file file.name "https://gateway.storjshare.io/yourbucketname/path/within/bucket?AWSAccessKeyId=jvruleqdpbwqx7vxmwgqbtlbmapa&Signature=fUNxawPyFd%2F9apR%2FZnKmR%2BPXGCA%3D&Expires=1628019103"

```

--------------------------------

### Create new Rails application for Storj integration

Source: https://storj.dev/dcs/code/rails-activestorage

Generates a new Rails 7 application named active-storj. Creates the complete project structure required for ActiveStorage integration with Storj. Requires Rails 7.0.4.2+ and Ruby 3.1.2+ installed on the system.

```shell
rails new active-storj
```

--------------------------------

### Configure JVM Options for Storj Storage

Source: https://storj.dev/dcs/third-party-tools/dataverse

Sets Payara JVM options for Storj S3 integration via asadmin. Requires Payara installed and running as user with AWS config. Enables S3-type storage with custom endpoint. Numerous options for fine-tuning; restart server to apply.

```bash
./asadmin create-jvm-options "\-Ddataverse.files.storj.type=s3"
./asadmin create-jvm-options "\-Ddataverse.files.storj.label=Storj"
./asadmin create-jvm-options "\-Ddataverse.files.storj.custom-endpoint-url=gateway.storjshare.io"
./asadmin create-jvm-options "\-Ddataverse.files.storj.custom-endpoint-region=us-1"
./asadmin create-jvm-options "\-Ddataverse.files.storj.bucket-name=dataverse"
./asadmin create-jvm-options "\-Ddataverse.files.storj.profile=default"
./asadmin create-jvm-options "\-Ddataverse.files.storj.path-style-access=false"
./asadmin create-jvm-options "\-Ddataverse.files.storj.payload-signing=false"
./asadmin create-jvm-options "\-Ddataverse.files.storj.upload-redirect=true"
./asadmin create-jvm-options "\-Ddataverse.files.storj.chunked-encoding=true"
./asadmin create-jvm-options "\-Ddataverse.files.storj.download-redirect=true"
./asadmin create-jvm-options "\-Ddataverse.files.storj.min-part-size=67198864"
./asadmin create-jvm-options "\-Ddataverse.files.storj.url-expiration-minutes=120"
```

--------------------------------

### Install Cyberduck on macOS via Homebrew

Source: https://storj.dev/dcs/third-party-tools/cyberduck

Installs the Cyberduck application on macOS using the Homebrew package manager. Ensure Homebrew is installed on your system before running this command.

```bash
brew install duck
```

--------------------------------

### Install Cyberduck on Windows via Chocolatey

Source: https://storj.dev/dcs/third-party-tools/cyberduck

Installs the Cyberduck application on Windows using the Chocolatey package manager. Ensure Chocolatey is installed on your system before running this command.

```powershell
choco install cyberduck
```

--------------------------------

### Create Storj Bucket

Source: https://storj.dev/dcs/third-party-tools/livepeer

Creates a new bucket named 'livepeer' on the Storj network using the uplink CLI. This bucket will be used to store input and output files for video processing.

```bash
uplink mb sj://livepeer
```

--------------------------------

### Install Cyberduck on Linux (RPM) using Yum

Source: https://storj.dev/dcs/third-party-tools/cyberduck

Installs Cyberduck on Linux distributions that use the Yum package manager. This involves adding the Cyberduck stable repository and then installing the 'duck' package.

```bash
echo -e "[duck-stable]\nname=duck-stable\nbaseurl=https://repo.cyberduck.io/stable/$basearch/\nenabled=1\ngpgcheck=0" | sudo tee /etc/yum.repos.d/duck-stable.repo

sudo yum install duck
```

--------------------------------

### Generate DNS zone file for Storj static site

Source: https://storj.dev/dcs/code/static-site-hosting

Shows the DNS zone file produced by the Uplink CLI, containing CNAME and TXT records needed to point a custom domain to the Storj static site. Replace <hostname>, <bucket>, <prefix>, and <access key> with your actual values. Adjust $ORIGIN and $TTL as required for your domain.

```dns
$ORIGIN example.com.
$TTL    3600
<hostname>	IN	CNAME	link.storjshare.io.
txt-<hostname>	IN	TXT	storj-root:<bucket>/<prefix>
txt-<hostname>	IN	TXT	storj-access:<access key>
txt-<hostname>	IN	TXT	storj-tls:true
```

```dns
$ORIGIN example.com.
$TTL    3600
www.example.com	IN	CNAME	link.storjshare.io.
txt-www.example.com	IN	TXT	storj-root:bucket/prefix
txt-www.example.com	IN	TXT	storj-access:abcdefghijklmnopqrstuvwxzy
txt-www.example.com	IN	TXT	storj-tls:true
```

--------------------------------

### Deploy Go Lambda Function with AWS CLI

Source: https://storj.dev/dcs/code/presigned-urls-serverless-cloud

Builds a Go application, packages it into a zip file, creates an IAM role for Lambda execution, and deploys the function. It also configures Lambda Function URL for direct invocation. Requires AWS CLI and Go installed.

```bash
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o main main.go  

zip lambda-handler.zip main  

aws iam create-role --role-name lambda-ex --assume-role-policy-document file://trust-policy.json  

aws iam attach-role-policy --role-name lambda-ex --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole  

aws lambda create-function --function-name cloudsigning --runtime go1.x --role arn:aws:iam::<YOUR IAM NUMBER>:role/lambda-ex --handler main --zip-file fileb://lambda-handler.zip  

aws lambda add-permission --function-name cloudsigning --action lambda:InvokeFunctionUrl --principal "*" --function-url-auth-type "NONE" --statement-id url  

aws lambda create-function-url-config --function-name cloudsigning --auth-type NONE  

curl 'https://<YOUR LAMBDA NUMBER>.lambda-url.us-east-1.on.aws/?key=test&method=POST'
```
Tokens:
Raw