S3Proxy

https://github.com/gaul/s3proxy
Admin
S3Proxy implements the S3 API and proxies requests to enable translation between S3 and various...
Tokens:165,948
Snippets:954
Trust Score:9.7
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start s3proxy with multiple properties files

Source: https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator

Command to start s3proxy with two properties files, enabling the alias blobstore configuration.

```bash
s3proxy --properties file1.conf --properties file2.conf
```

--------------------------------

### Start s3proxy service

Source: https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override

Command to start the s3proxy service with the specified properties file.

```bash
./s3proxy --properties s3proxy.properties
```

--------------------------------

### SFTP storage mapping example

Source: https://github.com/gaul/s3proxy/blob/master/docs/SFTP.md

Example showing how a bucket and object key map to an SFTP path. The bucket name becomes part of the SFTP path, so S3-compatible backup clients should configure the client bucket name to the desired directory name under s3proxy.sftp.basedir.

```properties
s3proxy.sftp.basedir=/data/backups
bucket=example-bucket
object-key=backups/app/1000/db_dump.zip
```

```text
/data/backups/example-bucket/backups/app/1000/db_dump.zip
```

--------------------------------

### SFTP backend configuration example

Source: https://github.com/gaul/s3proxy/blob/master/docs/SFTP.md

Configuration example for the SFTP backend. Sets S3Proxy client credentials and SFTP server credentials separately, pins the SFTP host-key fingerprint, and sets the base directory. If jclouds.endpoint omits a port, SFTP port 22 is used; the default s3proxy.sftp.basedir is /s3proxy.

```properties
s3proxy.authorization=aws-v2-or-v4
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.identity=local-identity
s3proxy.credential=local-credential

jclouds.provider=sftp
jclouds.endpoint=sftp://127.0.0.1:2222/
jclouds.identity=sftp-user
jclouds.credential=sftp-password
s3proxy.sftp.basedir=/s3proxy
s3proxy.sftp.host-key=SHA256:...
```

--------------------------------

### libcloud S3 driver setup

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Create a libcloud S3 driver with local credentials and non-secure connection.

```python
from libcloud.storage.types import Provider
from libcloud.storage.providers import get_driver
cls = get_driver(Provider.S3)
driver = cls('local-identity', 'local-credential', host='localhost', port=8080, secure=False)
```

--------------------------------

### AWS SDK for Java S3 client setup

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Create an AmazonS3Client with basic credentials, signature v2 override, and path-style access.

```java
AmazonS3Client client = new AmazonS3Client(
    new BasicAWSCredentials("identity", "credential"),
    new ClientConfiguration().withSignerOverride("S3SignerType"));
client.setEndpoint("http://127.0.0.1:8080");
client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));
```

--------------------------------

### Example: expose scoped-data/ prefix as customer-bucket

Source: https://github.com/gaul/s3proxy/wiki/Middleware-prefix-blobstore

Configures the customer-bucket to map onto the scoped-data/ prefix in backend storage. All reads, writes, listings, and multipart uploads to customer-bucket operate under that prefix.

```properties
s3proxy.prefix-blobstore.customer-bucket=scoped-data/
```

--------------------------------

### Regex rename example for type_partition

Source: https://github.com/gaul/s3proxy/wiki/Middleware-regex

Example rule that rewrites a key like prefix/test/2023/01/01/test.txt to prefix/date=2023-01-01/type=test/test.txt. The match expression must use escaped backslashes (e.g., \d) because it is in escaped string format.

```properties
s3proxy.regex-blobstore.match.type_partition=^prefix/(\w+)/(\d{4})/(\d{2})/(\d{2})/(.*)$
s3proxy.regex-blobstore.replace.type_partition=prefix/date=$2-$3-$4/type=$1/$5
```

--------------------------------

### Input key name for regex rename

Source: https://github.com/gaul/s3proxy/wiki/Middleware-regex

Example input key name that matches the type_partition rule.

```text
prefix/test/2023/01/01/test.txt
```

--------------------------------

### Run AwsSdkTest against SFTP backend

Source: https://github.com/gaul/s3proxy/blob/master/docs/SFTP.md

Runs the full AwsSdkTest suite against the SFTP backend in CI. Uses the s3proxy-sftp.conf test configuration and the embedded SftpTestServerListener, so no external setup is needed.

```bash
mvn test -Ds3proxy.test.conf=s3proxy-sftp.conf -Dtest=AwsSdkTest
```

--------------------------------

### Configure simulated latency and speed in S3Proxy

Source: https://github.com/gaul/s3proxy/wiki/Middleware-latency

Configuration properties for S3Proxy to simulate latency and speed. Use wildcard '*' to apply to all operations, or specify an operation name (e.g., 'put', 'get') to override for that operation. Latency is in milliseconds, speed in kb/s.

```properties
s3proxy.latency-blobstore.*.latency=1000 # use 1000ms latency for all operations
s3proxy.latency-blobstore.put.latency=2000 # use 2000ms latency for put blob instead

s3proxy.latency-blobstore.*.speed=100 # use 100 kb/s for all operations
s3proxy.latency-blobstore.get.speed=10 # use 10 kb/s for get blob instead
```

--------------------------------

### Get SFTP host-key fingerprint with ssh-keyscan

Source: https://github.com/gaul/s3proxy/blob/master/docs/SFTP.md

Command to obtain an OpenSSH-format SHA256 host-key fingerprint for the SFTP server, typically used to set s3proxy.sftp.host-key.

```bash
ssh-keyscan -p 2222 127.0.0.1 | ssh-keygen -lf -
```

--------------------------------

### s3cmd .s3cfg sample

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Sample .s3cfg configuration for s3cmd 1.5.0 and newer.

```ini
[default]
access_key = local-identity
secret_key = local-credential
host_base = localhost:8080
host_bucket = localhost:8080
```

--------------------------------

### Test S3Proxy by creating and listing buckets

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Creates a bucket named testbucket using curl and then lists all buckets to verify. The second command includes the S3 XML response showing the newly created bucket.

```bash
curl --request PUT http://localhost:8080/testbucket
```

```bash
curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>
```

--------------------------------

### s3cmd .s3cfg sample

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Sample configuration for s3cmd before 1.5.0. Sets access key, secret key, host_base to localhost:8080, and host_bucket pattern.

```ini
[default]
access_key = local-identity
secret_key = local-credential
host_base = localhost:8080
host_bucket = %(bucket)s.localhost:8080
```

--------------------------------

### Deploy S3Proxy with kubectl apply

Source: https://github.com/gaul/s3proxy/blob/master/examples/kubernetes/README.md

Apply the Kubernetes manifests in the cloud-backend directory to deploy S3Proxy. The manifests are minimal and unparameterized; copy and adjust them for your configuration management.

```bash
kubectl apply -f cloud-backend/
```

--------------------------------

### AWS SDK for .NET (AWSSDK.Extensions.NETCore.Setup) S3 client registration

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Register an IAmazonS3 client for dependency injection, using AnonymousAWSCredentials and path-style access.

```csharp
builder.Services.AddSingleton<IAmazonS3>(_ =>
{
    var s3Config = new AmazonS3Config
    {
        ServiceURL = "http://localhost:8080",
        ForcePathStyle = true,
        UseHttp = storageOptions.ServiceUrl.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
    };
    return new AmazonS3Client(new AnonymousAWSCredentials(), s3Config);
});
```

--------------------------------

### Run S3Proxy with trace logging

Source: https://github.com/gaul/s3proxy/wiki/Tips

Run S3Proxy with trace logging to show client requests and server responses.

```bash
java -DLOG_LEVEL=trace -jar s3proxy.jar
```

--------------------------------

### Configure S3Proxy eventual consistency with two backends

Source: https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency

Configures S3Proxy to simulate eventual consistency by directing writes to the primary backend and reads to the alternate backend. The delay is in seconds and the probability is a number between 0.0 and 1.0. Requires two storage backends: the primary (jclouds) and the alternate (alt.jclouds).

```properties
s3proxy.endpoint=http://localhost:8081
s3proxy.authorization=aws-v2
s3proxy.identity=local-identity
s3proxy.credential=local-credential
s3proxy.eventual-consistency=true
s3proxy.eventual-consistency.delay=5  # time in seconds
s3proxy.eventual-consistency.probability=1  # percentage expressed as a number between 0.0 and 1.0

jclouds.provider=transient
jclouds.identity=remote-identity
jclouds.credential=remote-credential

alt.jclouds.provider=transient
alt.jclouds.identity=remote-identity
alt.jclouds.credential=remote-credential
```

--------------------------------

### Run S3Proxy on Windows

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Runs S3Proxy on Windows by explicitly invoking java with the executable jar and the properties file s3proxy.conf.

```bash
java -jar s3proxy --properties s3proxy.conf
```

--------------------------------

### AWS CLI configuration and list buckets

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Configure AWS CLI with local credentials and list S3 buckets using the S3Proxy endpoint.

```bash
$ aws configure
AWS Access Key ID [****]: local-identity
AWS Secret Access Key [****]: local-credential
Default region name [None]:
Default output format [None]:

$ aws s3 ls --endpoint-url http://127.0.0.1:8080
```

--------------------------------

### Run S3Proxy with trace logging and jclouds wire debug

Source: https://github.com/gaul/s3proxy/wiki/Tips

Run S3Proxy with trace logging and jclouds wire debugging to emit storage backend requests and responses.

```bash
java -DLOG_LEVEL=trace -Djclouds.wire=debug -jar s3proxy.jar
```

--------------------------------

### fog storage configuration

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Create a Fog::Storage provider for AWS with path-style access.

```ruby
fog = Fog::Storage.new(provider: 'aws', endpoint: 'http://127.0.0.1:8080',
    path_style: true)
```

--------------------------------

### Create keystore with keytool

Source: https://github.com/gaul/s3proxy/wiki/SSL-support

Generates a keystore.jks holding a key pair with alias 'aws' using RSA. Use CN=*.s3.amazonaws.com to proxy Amazon S3; applications will reject the self-signed certificate unless imported into their trusted store.

```bash
$ keytool -keystore keystore.jks -alias aws -genkey -keyalg RSA
```

--------------------------------

### Create S3Proxy filesystem basedir

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Creates the filesystem base directory required by the filesystem backend configuration.

```bash
mkdir /tmp/s3proxy
```

--------------------------------

### Enable NoCacheBlobStore via system property

Source: https://github.com/gaul/s3proxy/wiki/Middleware-no-cache

Set the system property `s3proxy.no-cache-blobstore=true` in the S3Proxy configuration to enable the middleware.

```properties
s3proxy.no-cache-blobstore=true
```

--------------------------------

### Swift (Keystone v2.0) backend configuration

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using Swift with Keystone v2.0 authentication. All $OS_ variables can be retrieved from your tenant specific openrc.sh file.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=openstack-swift
jclouds.endpoint=$OS_AUTH_URL (e.g. https://keystone.example.com/v2.0/)
jclouds.region=$OS_REGION_NAME (e.g. paris1)
jclouds.identity=$OS_TENANT_NAME:$OS_USERNAME (e.g. production:user1)
jclouds.credential=$OS_PASSWORD (e.g. password)
```

--------------------------------

### Configure regex-blobstore match and replace rules

Source: https://github.com/gaul/s3proxy/wiki/Middleware-regex

Configuration template for defining multiple regex rename rules. Each match rule must be accompanied by its replace rule. Rules are evaluated in order, and the first match stops evaluation.

```properties
s3proxy.regex-blobstore.match.<regex name 1> = <regex match expression>
s3proxy.regex-blobstore.replace.<regex name 1> = <regex replace expression>

    .... [snip] ....

s3proxy.regex-blobstore.match.<regex name N> = <regex match expression>
s3proxy.regex-blobstore.replace.<regex name N> = <regex replace expression>
```

--------------------------------

### Configure bucket-locator with bucket names

Source: https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator

Specify bucket names for each backend in the properties file. The indexing (1, 2) differentiates entries, and glob syntax is supported for bucket names.

```properties
s3proxy.bucket-locator.1=bucket1
s3proxy.bucket-locator.2=bucket2
```

--------------------------------

### Swift (non-Keystone) backend configuration

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using Swift without Keystone authentication. Marked as 'outdated?!' in the source.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=swift
jclouds.endpoint=http://IP-OF-SWIFT:PORT/auth/v1.0
jclouds.api=swift
jclouds.identity=test:tester
jclouds.credential=testing
```

--------------------------------

### Swift (Keystone v3.0) backend configuration

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using Swift with Keystone v3.0 authentication. All $OS_ variables can be retrieved from your project specific openrc.sh file.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=openstack-swift
jclouds.endpoint=$OS_AUTH_URL (e.g. https://keystone.example.com/v3/)
jclouds.regions=$OS_REGION_NAME (e.g. paris1)
jclouds.region=$OS_REGION_NAME (e.g. paris1)
jclouds.identity=$OS_USER_DOMAIN_NAME:$OS_USERNAME (e.g. domain1:user1)
jclouds.keystone.version=3
jclouds.keystone.scope=project:$OS_PROJECT_NAME (e.g. project:project1)
jclouds.keystone.project_domain_name=$OS_PROJECT_DOMAIN_NAME (e.g. domain2 - only needed if user and project domains differ)
jclouds.credential=$OS_PASSWORD (e.g. password)
```

--------------------------------

### Run S3Proxy on Linux/Mac

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Runs S3Proxy on Linux and Mac OS X by making the executable jar runnable and then invoking it with the properties file s3proxy.conf.

```bash
chmod +x s3proxy
s3proxy --properties s3proxy.conf
```

--------------------------------

### Enable prefix middleware for a bucket

Source: https://github.com/gaul/s3proxy/wiki/Middleware-prefix-blobstore

Add one property per bucket that should be scoped to a prefix. Replace <bucket-name> with the bucket name and <prefix> with the desired prefix.

```properties
s3proxy.prefix-blobstore.<bucket-name>=<prefix>
```

--------------------------------

### boto3 session and S3 client/resource

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Create a boto3 session with path-style addressing and use it for low-level client or resource.

```python
session = boto3.session.Session(aws_access_key_id='identity',
                                aws_secret_access_key='credential')
config = boto3.session.Config(s3={'addressing_style': 'path'})
# low level S3 client
client = session.client('s3', endpoint_url='http://localhost:8080',
                        config=config)
# S3 resource object
resource = session.resource('s3', endpoint_url='http://localhost:8080',
                            config=config)
```

--------------------------------

### Test storage class override with AWS CLI

Source: https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override

Commands to set credentials, create a test file, upload it to S3, and check the object's storage class using the AWS CLI against the proxy endpoint.

```bash
export AWS_ACCESS_KEY_ID=local-identity
export AWS_SECRET_ACCESS_KEY=local-credential
echo "put me in test" > test.txt
aws s3 --endpoint-url http://127.0.0.1:8080 cp test.txt s3://BUCKETNAME/test.txt
aws s3api head-object --endpoint-url http://127.0.0.1:8080 --bucket BUCKETNAME --key test.txt
```

--------------------------------

### Enable NoCacheBlobStore via Docker environment variable

Source: https://github.com/gaul/s3proxy/wiki/Middleware-no-cache

Set the environment variable `S3PROXY_NO_CACHE_BLOBSTORE=true` for the Docker container to enable the middleware.

```bash
S3PROXY_NO_CACHE_BLOBSTORE=true
```

--------------------------------

### Run S3Proxy with debug logging

Source: https://github.com/gaul/s3proxy/wiki/Tips

Run S3Proxy with debug logging to reveal more information about requests.

```bash
java -DLOG_LEVEL=debug -jar s3proxy.jar
```

--------------------------------

### Add bucket.localhost to /etc/hosts

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Ensure bucket.localhost resolves to localhost by adding this line to /etc/hosts.

```hosts
127.0.0.1       bucket.localhost
```

--------------------------------

### jclouds-cli blobstore container-list

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

List containers using jclouds-cli with S3 API and S3Proxy endpoint.

```bash
jclouds blobstore container-list --api s3 \
    --endpoint http://127.0.0.1:8080 \
    --identity local-identity \
    --credential local-credential
```

--------------------------------

### S3-compatible backend configuration (DreamObjects)

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using a generic S3-compatible provider (e.g., DreamObjects). Note that jclouds uses AWSSIGV2 by default, which may not be supported by some providers; set jclouds.s3.signer-version=4 to override.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=s3
jclouds.identity=xxxxxxxxxxxxxxxxxxxx
jclouds.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
jclouds.endpoint=https://objects.dreamhost.com
```

--------------------------------

### Configuration for user-metadata-replacer middleware

Source: https://github.com/gaul/s3proxy/wiki/Middleware-user-metadata-replacer

Enables the middleware and specifies the character replacement: from '_' to '-'. Set these properties in the s3proxy configuration file.

```properties
s3proxy.user-metadata-replacer-blobstore=true
s3proxy.user-metadata-replacer-blobstore.from-chars=_
s3proxy.user-metadata-replacer-blobstore.to-chars=-
```

--------------------------------

### Configure sharded blobstore middleware

Source: https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend

Use this configuration to enable sharding for a virtual bucket named 'sharded-bucket' with 10 shards and a 'shard' prefix. When the prefix option is omitted, the virtual bucket name is used as the shard prefix. CreateBucket creates all shards; DeleteBucket removes them.

```properties
s3proxy.sharded-blobstore.sharded-bucket.shards=10
s3proxy.sharded-blobstore.sharded-bucket.prefix=shard
```

--------------------------------

### Configure bucket-to-backend assignment with s3proxy.bucket-locator

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Assigns buckets to different backends using the s3proxy.bucket-locator properties. Each property key (e.g., s3proxy.bucket-locator.1) specifies a bucket name or glob pattern to route to a specific backend. A bucket or glob cannot be assigned to multiple backends.

```properties
s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket
```

--------------------------------

### SoftLayer backend configuration (Dallas geo)

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using SoftLayer in the Dallas geo, which uses Keystone with tempAuthCredentials.

```properties
s3proxy.endpoint=http://localhost:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=openstack-swift
jclouds.endpoint=https://dal05.objectstorage.softlayer.net/auth/v1.0
jclouds.identity=test:tester
jclouds.credential=testing
jclouds.keystone.credential-type=tempAuthCredentials
```

--------------------------------

### Maven repository and dependency for S3Proxy snapshot

Source: https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects

Configures the Sonatype snapshots repository and adds the S3Proxy 2.9.0-SNAPSHOT dependency for pre-release versions.

```xml
<repositories>
    <repository>
        <id>sonatype-snapshots</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

...

<dependency>
    <groupId>org.gaul</groupId>
    <artifactId>s3proxy</artifactId>
    <version>2.9.0-SNAPSHOT</version>
</dependency>
```

--------------------------------

### Export and import self-signed certificate for Java applications

Source: https://github.com/gaul/s3proxy/wiki/SSL-support

Exports the certificate from the keystore and imports it into the Java truststore so Java applications accept the self-signed certificate. Run the export first, then the import.

```bash
$ keytool -exportcert -keystore keystore.jks -alias aws -rfc > aws.crt
$ keytool -keystore $JAVA_HOME/jre/lib/security/cacerts -import -alias aws -file aws.crt -trustcacerts
```

--------------------------------

### Configure Azure Blob key backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use Azure Blob storage with static key authorization. The jclouds.azureblob.auth is set to azureKey and the provider is azureblob. Only static key authorization is supported by this backend.

```properties
s3proxy.endpoint=http://0.0.0.0:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.azureblob.auth=azureKey
jclouds.provider=azureblob
jclouds.identity=xxx
jclouds.credential=yyyyyyyyyyyyy
jclouds.endpoint=https://xxx.blob.core.windows.net

```

--------------------------------

### Create s3proxy.properties configuration file

Source: https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override

Configuration for s3proxy to use the storage-class-blobstore middleware with STANDARD_IA. Sets endpoint, authorization, identity, credential, and jclouds provider settings for AWS S3.

```properties
s3proxy.storage-class-blobstore=STANDARD_IA
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=aws-s3
jclouds.identity=AKIAXXXXXXXXXXXXXXXXXX
jclouds.credential=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
```

--------------------------------

### Configure Backblaze B2 backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use Backblaze B2 as the backend. Set the endpoint, authorization, and credentials. The provider is b2 and the identity and credential are the Backblaze B2 application key ID and application key.

```properties
s3proxy.endpoint=http://127.0.0.1:8081
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=b2
jclouds.identity=xxxxxxxxxxxx
jclouds.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

```

--------------------------------

### AWS SDK for Ruby configuration

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Configure the AWS SDK for Ruby with credentials and endpoint for S3Proxy.

```ruby
require 'aws-sdk'

Aws.config.update({
    credentials: Aws::Credentials.new('identity', 'credential'),
    endpoint: 'http://localhost:8080'
})
```

--------------------------------

### AWS SDK for Hadoop/Spark S3A configuration

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Configure Hadoop S3A filesystem to use S3Proxy, with options for user/password or anonymous access. Requires setting s3proxy.authorization=aws-v2-or-v4 with latest hadoop-aws jar.

```java
sc.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
sc.hadoopConfiguration.set("fs.s3a.endpoint", "127.0.0.1:4242")
sc.hadoopConfiguration.set("fs.s3a.connection.ssl.enabled", "false") // if no SSL support
// if user/password access
sc.hadoopConfiguration.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
sc.hadoopConfiguration.set("fs.s3a.access.key", "local-identity")
sc.hadoopConfiguration.set("fs.s3a.secret.key", "local-credential")
// if anonymous access
sc.hadoopConfiguration.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider")
```

--------------------------------

### Configure S3Proxy with filesystem backend

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Configures S3Proxy to use the local filesystem as the storage backend with anonymous access. The endpoint is set to http://127.0.0.1:8080 and the base directory is /tmp/s3proxy.

```properties
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy
```

--------------------------------

### Output of head-object showing StorageClass

Source: https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override

JSON output from the head-object command showing the object was created with the target storage class STANDARD_IA.

```json
{
    "LastModified": "2024-07-21T17:00:44+00:00",
    "ContentLength": 15,
    "ETag": "\"a99a4b4b9e8204c1489f9f37dd21ec1f\"",
    "ContentEncoding": "",
    "ContentType": "text/plain",
    "Metadata": {},
    "StorageClass": "STANDARD_IA"
}
```

--------------------------------

### s3fs-fuse mount command

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Mount an s3proxy bucket using s3fs-fuse with sigv2 and path request style.

```bash
$ echo local-identity:local-credential > passwd
$ chmod 600 passwd
$ s3fs bucketname dirname \
      -o passwd_file=passwd \
      -o sigv2 \
      -o url=http://127.0.0.1:8080 \
      -o use_path_request_style
```

--------------------------------

### Output key name after regex rename

Source: https://github.com/gaul/s3proxy/wiki/Middleware-regex

Rewritten key name produced by the type_partition rule.

```text
prefix/date=2023-01-01/type=test/test.txt
```

--------------------------------

### SFTP path mapping pattern

Source: https://github.com/gaul/s3proxy/blob/master/docs/SFTP.md

Effective path mapping for the SFTP backend: S3 buckets are first-level directories below s3proxy.sftp.basedir, and object keys map to files below the bucket directory.

```text
<s3proxy.sftp.basedir>/<bucket>/<object-key>
```

--------------------------------

### Configure AWS S3 backend credentials with session token

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Configures the S3Proxy storage backend with AWS S3 credentials, including an optional session token for expiring credentials. This properties-style configuration is read at startup and can be used with jclouds.identity, jclouds.credential, and jclouds.session-token.

```properties
jclouds.provider=aws-s3
jclouds.identity=ASIAIOSFODNN7EXAMPLE
jclouds.credential=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
jclouds.session-token=IQoJb3JpZ2luX2VjEExample
```

--------------------------------

### Configure S3Proxy secure-endpoint without Docker

Source: https://github.com/gaul/s3proxy/wiki/SSL-support

Set the secure-endpoint, keystore path, and keystore password in the S3Proxy configuration to enable HTTPS without Docker.

```properties
s3proxy.secure-endpoint=https://0.0.0.0:443
s3proxy.keystore-path=keystore.jks
s3proxy.keystore-password=password
```

--------------------------------

### Google Cloud Storage backend configuration

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using Google Cloud Storage as the backend. The jclouds.credential value is a placeholder for a private key, with line continuation backslashes shown in the source.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=google-cloud-storage
jclouds.identity=999999999999-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com
jclouds.credential=jclouds.credential=-----BEGIN RSA PRIVATE KEY-----\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\
-----END RSA PRIVATE KEY-----
```

--------------------------------

### Configure Google Cloud Storage file-based backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use Google Cloud Storage with a file-based service account key. The jclouds.credential points to a file containing the RSA private key. The identity is the service account email. The second block shows the expected content of that credentials file.

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=google-cloud-storage
jclouds.identity=999999999999-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com
jclouds.credential=/path/to/credentials.key

```

```text
-----BEGIN RSA PRIVATE KEY-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END RSA PRIVATE KEY-----

```

--------------------------------

### Configure AWS v4 authentication in properties file

Source: https://github.com/gaul/s3proxy/wiki/Tips

Configure S3Proxy to require AWS Signature Version 4 authentication. Replace <access-key> and <secret-key> with your credentials. Misconfiguration can cause spurious client failures, so ensure NTP is enabled on all machines.

```properties
s3proxy.authorization=aws-v4
s3proxy.identity=<access-key>
s3proxy.credential=<secret-key>
```

--------------------------------

### dnsmasq wildcard localhost resolution

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Command to make *.localhost resolve to 127.0.0.1 using dnsmasq with NetworkManager.

```bash
echo address=/localhost/127.0.0.1 | sudo tee /etc/NetworkManager/dnsmasq.d/localhost.conf
```

--------------------------------

### First properties file for alias blobstore disambiguation

Source: https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator

First properties file for a backend that uses alias blobstore to disambiguate a bucket named 'data' that exists in two backends. Maps alias 'data-backend-1' to 'data' and sets bucket-locator to that alias.

```properties
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/one
s3proxy.alias-blobstore.data-backend-1=data
s3proxy.bucket-locator.1=data-backend-1
```

--------------------------------

### Rackspace Cloud Files backend configuration with region

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configuration for using Rackspace Cloud Files with a user-specified region (HKG).

```properties
s3proxy.endpoint=http://127.0.0.1:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=rackspace-cloudfiles-us
jclouds.identity=xxxxxxxxxxxxxx
jclouds.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
jclouds.region=HKG
```

--------------------------------

### Second properties file for alias blobstore disambiguation

Source: https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator

Second properties file for the other backend, using alias 'data-backend-2' for the same bucket 'data'. This allows clients to access the bucket in both backends with the same s3proxy credentials.

```properties
s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/two
s3proxy.alias-blobstore.data-backend-2=data
s3proxy.bucket-locator.1=data-backend-2
```

--------------------------------

### s3proxy.properties virtual-host setting

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Add this to s3proxy.properties to enable virtual-host style requests for s3cmd.

```properties
s3proxy.virtual-host=localhost
```

--------------------------------

### boto connection to S3Proxy

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Connect to S3Proxy using boto with ordinary calling format and non-SSL.

```python
boto.connect_s3(aws_access_key_id = 'identity',
    aws_secret_access_key = 'credential',
    host = '127.0.0.1', port = 8080, is_secure = False,
    calling_format = boto.s3.connection.OrdinaryCallingFormat())
```

--------------------------------

### Configure AWS S3 SDK backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use the AWS SDK backend for S3-compatible storage. Set the endpoint, authorization, and credentials. The jclouds.provider is aws-s3-sdk and the region is us-east-1. The conditional-writes option can be set to 'emulated' if the backend does not support native conditional writes.

```properties
s3proxy.endpoint=http://0.0.0.0:8080 (host:port s3proxy is available at)
s3proxy.authorization=aws-v2-or-v4
# identity and credential correspond to AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY that you use to authenticate to s3proxy
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=aws-s3-sdk
# change to any s3 compatible storage
jclouds.endpoint=https://s3.us-east-1.amazonaws.com 
aws-s3-sdk.region=us-east-1
# change to emulated if you need the feature and your backend doesn't support it
aws-s3-sdk.conditional-writes=native 

```

--------------------------------

### S3Proxy large object mocking properties file

Source: https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking

Configuration properties to enable null blobstore for mocking large objects. Sets endpoint, authorization, identity, credential, and null-blobstore flag, along with jclouds provider and credentials.

```properties
s3proxy.endpoint=http://localhost:8081
s3proxy.authorization=aws-v2
s3proxy.identity=local-identity
s3proxy.credential=local-credential
s3proxy.null-blobstore=true

jclouds.provider=transient
jclouds.identity=remote-identity
jclouds.credential=remote-credential
```

--------------------------------

### AWS SDK for JavaScript in node.js S3 client

Source: https://github.com/gaul/s3proxy/wiki/Client-compatibility-list

Create an S3 client with endpoint, credentials, and path-style access. Also specify jclouds.regions = us-east-1 to work around aws/aws-sdk-js#919.

```javascript
var AWS = require('aws-sdk');
var s3 = new AWS.S3({
  endpoint: 'http://127.0.0.1:8080',
  accessKeyId: 'local-identity',
  secretAccessKey: 'local-credential',
  s3ForcePathStyle: true
});
```

--------------------------------

### S3Proxy read-only properties file

Source: https://github.com/gaul/s3proxy/wiki/Middleware-read-only

Use this properties file to run S3Proxy in read-only mode, blocking mutating operations like put object. The key setting is s3proxy.read-only-blobstore=true, which must be combined with the other required S3Proxy and jclouds properties shown.

```properties
s3proxy.endpoint=http://localhost:8081
s3proxy.authorization=aws-v2
s3proxy.identity=local-identity
s3proxy.credential=local-credential
s3proxy.read-only-blobstore=true

jclouds.provider=transient
jclouds.identity=remote-identity
jclouds.credential=remote-credential
```

--------------------------------

### Configure Azure Blob SDK managed identity backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use the Azure Blob SDK backend with managed identity. If jclouds.identity and jclouds.credential are empty strings, DefaultAzureCredential is used for authentication. Setting these options disables managed identity and uses a static key instead.

```properties
s3proxy.endpoint=http://0.0.0.0:8080
s3proxy.authorization=none
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=azureblob-sdk
jclouds.endpoint=https://xxx.blob.core.windows.net
jclouds.identity=
jclouds.credential=

```

--------------------------------

### Configure CORS in S3Proxy properties with s3proxy.cors-allow-origins

Source: https://github.com/gaul/s3proxy/blob/master/README.md

Configures CORS preflight and actual request/response handling in the properties file (and corresponding ENV variables for Docker). The origins property uses a space-separated list of regular expressions; allowed methods and headers are space-separated. The last line enables credentials.

```properties
s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true
```

--------------------------------

### Maven dependency for S3Proxy release

Source: https://github.com/gaul/s3proxy/wiki/Using-S3Proxy-in-Java-projects

Adds the stable S3Proxy 2.8.0 release to a Maven project.

```xml
<dependency>
    <groupId>org.gaul</groupId>
    <artifactId>s3proxy</artifactId>
    <version>2.8.0</version>
</dependency>
```

--------------------------------

### Configure AWS S3 managed identity backend

Source: https://github.com/gaul/s3proxy/wiki/Storage-backend-examples

Configure S3Proxy to use AWS S3 with managed identity via IAM roles for service accounts. Set the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE to enable automatic role assumption. The jclouds.identity and jclouds.credential are left empty to use the web identity credential provider.

```properties
s3proxy.endpoint=http://0.0.0.0:8080
s3proxy.authorization=aws-v2-or-v4
s3proxy.identity=local-identity
s3proxy.credential=local-credential
jclouds.provider=aws-s3
jclouds.endpoint=https://s3.xxx.amazonaws.com
jclouds.identity=
jclouds.credential=

```

--------------------------------

### Disable S3Proxy authentication

Source: https://github.com/gaul/s3proxy/wiki/Tips

Disable S3Proxy authentication as a last resort when authentication misconfiguration causes client failures. This is useful for debugging with curl.

```properties
s3proxy.authorization=none
```
Tokens:
Raw