Remotefs Aws S3

https://docs.rs/remotefs-aws-s3/0.4.3
Admin
Remotefs Aws S3 is a client implementation for remotefs, providing support for the AWS S3 protocol,...
Tokens:333,910
Snippets:1,522
Trust Score:9.7
Update:1 day ago
Show doc for...
Tokens:
Raw
### Create and use a MinIO client with AwsS3Fs

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/

Shows how to instantiate an AwsS3Fs client for a MinIO server, using endpoint and new_path_style(true) which is required for MinIO, then connect, get working directory, change directory, and disconnect. This example is not tested.

```rust
use remotefs::RemoteFs;
use remotefs_aws_s3::AwsS3Fs;
use std::path::Path;

let mut client = AwsS3Fs::new("test-bucket", &Arc::new(tokio::runtime::Runtime::new().unwrap()))
    .endpoint("http://localhost:9000")
    .new_path_style(true) // required for MinIO
    .access_key("minioadmin")
    .secret_access_key("minioadmin");
// connect
assert!(client.connect().is_ok());
// get working directory
println!("Wrkdir: {}", client.pwd().ok().unwrap().display());
// change working directory
assert!(client.change_dir(Path::new("/tmp")).is_ok());
// disconnect
assert!(client.disconnect().is_ok());
```

--------------------------------

### Create and use an AWS S3 client with AwsS3Fs

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/

Shows how to instantiate an AwsS3Fs client for an AWS S3 bucket, configure region, profile, and credentials, then connect, get working directory, change directory, and disconnect. This example is not tested.

```rust
use remotefs::RemoteFs;
use remotefs_aws_s3::AwsS3Fs;
use std::path::Path;

let mut client = AwsS3Fs::new("test-bucket", &Arc::new(tokio::runtime::Runtime::new().unwrap()))
    .region("eu-west-1")
    .profile("default")
    .access_key("AKIAxxxxxxxxxxxx")
    .secret_access_key("****************");
// connect
assert!(client.connect().is_ok());
// get working directory
println!("Wrkdir: {}", client.pwd().ok().unwrap().display());
// change working directory
assert!(client.change_dir(Path::new("/tmp")).is_ok());
// disconnect
assert!(client.disconnect().is_ok());
```

--------------------------------

### AbortMultipartUpload fluent builder example

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Shows how to call the abort_multipart_upload operation on an S3Client, setting the bucket to "example" and sending the request asynchronously. The result is an async future that returns a result.

```rust
let result = client.abort_multipart_upload()
    .bucket("example")
    .send()
    .await;
```

--------------------------------

### Customizing Client Configuration with aws_sdk_s3::config::Builder

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Shows how to customize service-specific settings on the Config using the Builder struct, which implements From<&SdkConfig>. The example includes a placeholder for a service-specific setting.

```rust
let sdk_config = ::aws_config::load_from_env().await;
let config = aws_sdk_s3::config::Builder::from(&sdk_config)
    .some_service_specific_setting("value")
    .build();
```

--------------------------------

### Initialize AwsS3Fs client and create bucket

Source: https://docs.rs/remotefs-aws-s3/0.4.3/src/remotefs_aws_s3/client.rs.html

Creates an AwsS3Fs client with MinIO endpoint and credentials, connects, creates a bucket named 'github-ci', creates a temporary working directory, and changes into it. Returns a Ctx struct with the client and container.

```rust
let mut client = AwsS3Fs::new("github-ci", &runtime)
    .endpoint(format!("http://localhost:{port}"))
    .access_key("minioadmin")
    .secret_access_key("minioadmin")
    .new_path_style(true);

// connect
assert!(client.connect().is_ok());

// Create bucket manually
let fut = client
    .client()
    .unwrap()
    .create_bucket()
    .bucket("github-ci")
    .send();
let res = runtime.block_on(fut);

assert!(res.is_ok());

// Create wrkdir
let tempdir = PathBuf::from(generate_tempdir());
assert!(
    client
        .create_dir(tempdir.as_path(), UnixPex::from(0o775))
        .is_ok()
);
// Change directory
assert!(client.change_dir(tempdir.as_path()).is_ok());
Ctx {
    client,
    container: minio,
}
```

--------------------------------

### pwd

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.AwsS3Fs.html

Gets the current working directory.

```APIDOC
## pwd

### Description
Get working directory.

### Method
pwd

### Signature
`fn pwd(&mut self) -> RemoteResult<PathBuf>`

### Parameters
None

### Returns
- `RemoteResult<PathBuf>` - On success, the current working directory as a PathBuf. On error, a RemoteError.

```

--------------------------------

### Constructing a Client with aws_config::load_from_env()

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Simplest way to create an S3 client. Requires the aws-config crate to resolve configuration from the environment. The client is created with aws_sdk_s3::Client::new(&config).

```rust
let config = aws_config::load_from_env().await;
let client = aws_sdk_s3::Client::new(&config);
```

--------------------------------

### abort_multipart_upload

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Example of using the abort_multipart_upload fluent builder to send a request.

```APIDOC
## abort_multipart_upload

### Description
Aborts a multipart upload. This method returns a fluent builder that can be configured and then sent asynchronously.

### Method
Client method (Rust)

### Endpoint
`S3Client::abort_multipart_upload`

### Parameters
- `bucket` (string) - Required - The bucket name.

### Request Example
```rust
let result = client.abort_multipart_upload()
    .bucket("example")
    .send()
    .await;
```

### Response
Returns an async future that resolves to a result.
```

--------------------------------

### client

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.AwsS3Fs.html

Get a reference to the underlying AWS SDK S3Client struct.

```APIDOC
## client

### Description
Get a reference to the underlying aws sdk [`S3Client`](https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html) struct.

### Method
`pub fn client(&self) -> Option<&S3Client>`

### Parameters
None.

### Returns
- **Option<&S3Client>** - A reference to the underlying S3Client if available, otherwise None.

### Example
```rust
let fs = AwsS3Fs::new();
if let Some(client) = fs.client() {
    // use client
}
```
```

--------------------------------

### AwsS3Fs::new

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.AwsS3Fs.html

Initializes a new AwsS3Fs instance with the specified bucket name and Tokio runtime.

```APIDOC
## AwsS3Fs::new

### Description
Initializes a new `AwsS3Fs` instance with the given bucket name and Tokio runtime.

### Method
`pub fn new<S: AsRef<str>>(bucket: S, runtime: &Arc<Runtime>) -> Self`

### Parameters
- **bucket** (S) - Required - The name of the S3 bucket.
- **runtime** (&Arc<Runtime>) - Required - A reference to a Tokio runtime.

### Returns
- **Self** - A new `AwsS3Fs` instance.
```

--------------------------------

### open

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.AwsS3Fs.html

Opens a file at the specified path for reading.

```APIDOC
## open

### Description
Opens a file at the specified path for reading.

### Method
fn open(&mut self, _path: &Path) -> RemoteResult<ReadStream>

### Parameters
- `_path` (&Path) - The path to the file.

### Returns
- `RemoteResult<ReadStream>` - A read stream for reading data.

```

--------------------------------

### HeadObjectOutput::content_range

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Returns the portion of the object returned in the response for a GET request.

```APIDOC
## HeadObjectOutput::content_range

### Description
Returns the portion of the object returned in the response for a GET request.

### Method
`content_range(&self) -> Option<&String>`

### Returns
- `Option<&String>` - The content range of the object.
```

--------------------------------

### SelectObjectContentFluentBuilder::expression_type / set_expression_type

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Sets the type of the provided expression (for example, SQL). This parameter is required.

```APIDOC
## expression_type(ExpressionType) / set_expression_type(Option<ExpressionType>)

### Description
Sets the type of the provided expression (for example, SQL). This parameter is required.

### Parameters
- **expression_type** (ExpressionType) - Required - The type of the provided expression (for example, SQL).
- **set_expression_type** (Option<ExpressionType>) - Required - The type of the provided expression (for example, SQL).

### Required
**true**
```

--------------------------------

### create_file implementation with multipart upload

Source: https://docs.rs/remotefs-aws-s3/0.4.3/src/remotefs_aws_s3/client.rs.html

Implements create_file for S3 by performing a multipart upload. It reads the input in chunks of MIN_MULTIPART_UPLOAD_SIZE, uploads each part, and completes the upload.

```rust
fn create_file(
    &mut self,
    path: &Path,
    _metadata: &Metadata,
    mut reader: Box<dyn Read + Send>,
) -> RemoteResult<u64> {
    self.check_connection()?;
    let src = self.resolve(path);
    let key = Self::fmt_path(src.as_path(), false);

    // we write 4096 bytes at a time
    debug!("Query PUT for key '{}'", key);
    let mut buf = vec![0; MIN_MULTIPART_UPLOAD_SIZE];
    let mut offset = 0;
    let mut part_number = 1;

    // create multipart upload
    let fut = self
        .client
        .as_ref()
        .unwrap()
        .create_multipart_upload()
        .bucket(self.bucket_name.as_str())
        .key(key.as_str())
        .send();

    let upload = self.runtime.block_on(fut).map_err(|e| {
        error!("Could not init multipart upload: {e:?}",);
        RemoteError::new_ex(
            RemoteErrorType::ProtocolError,
            format!("Could not init multipart upload: {}", e),
        )
    })?;
    let upload_id = upload.upload_id().ok_or(RemoteError::new_ex(
        RemoteErrorType::ProtocolError,
        "no multipart id",
    ))?;
    debug!("starting multipart upload with upload id {upload_id}");

    let mut upload_parts: Vec<aws_sdk_s3::types::CompletedPart> = Vec::new();

    loop {
        let buflen = reader.read(&mut buf).map_err(|e| {
            RemoteError::new_ex(
                RemoteErrorType::IoError,
                format!("Could not read file: {}", e),
            )
        })?;
        if buflen == 0 {
            break;
        }

        let bytestream = ByteStream::new(SdkBody::from(&buf[..buflen]));
        debug!("sending part {part_number} for {key} at offset {offset} for {buflen} bytes");

        let buflen = buflen as i64;

        let fut = self
            .client
            .as_ref()
            .unwrap()
            .upload_part()
            .bucket(self.bucket_name.as_str())
            .key(key.as_str())
            .upload_id(upload_id)
            .body(bytestream)
            .part_number(part_number)
            .send();

        let upload_part_res = self.runtime.block_on(fut).map_err(|e| {
            error!("Could not put file: {e:?}",);
            RemoteError::new_ex(
                RemoteErrorType::ProtocolError,
                format!("Could not put file: {}", e),
            )
        })?;

        upload_parts.push(
            CompletedPart::builder()
                .e_tag(upload_part_res.e_tag.unwrap_or_default())
                .part_number(part_number)
                .build(),
        );

        debug!("written {buflen} bytes at offset {offset} for {key}");

        offset += buflen;
        part_number += 1;
    }

    // complete multipart
    let completed_multipart_upload: CompletedMultipartUpload =
        CompletedMultipartUpload::builder()
            .set_parts(Some(upload_parts))
            .build();

    let fut = self
        .client
        .as_ref()
        .unwrap()
        .complete_multipart_upload()
        .bucket(self.bucket_name.as_str())
        .key(key.as_str())
        .upload_id(upload_id)
        .multipart_upload(completed_multipart_upload)
        .send();
}
```

--------------------------------

### config

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Returns the client's configuration.

```APIDOC
## config

### Description
Returns the client's configuration.

### Method
`pub fn config(&self) -> &Config`

### Parameters
None.

### Returns
A reference to the client's `Config`.
```

--------------------------------

### SelectObjectContentFluentBuilder::scan_range / set_scan_range

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Specifies the byte range of the object to get the records from. This parameter is optional.

```APIDOC
## scan_range(ScanRange) / set_scan_range(Option<ScanRange>)

### Description
Specifies the byte range of the object to get the records from. A record is processed when its first byte is contained by the range. This parameter is optional, but when specified, it must not be empty. See RFC 2616, Section 14.35.1 about how to specify the start and end of the range.

### Parameters
- **scan_range** (ScanRange) - Optional - Specifies the byte range of the object to get the records from.
- **set_scan_range** (Option<ScanRange>) - Optional - Specifies the byte range of the object to get the records from.

### Required
**false**
```

--------------------------------

### ListBucketMetricsConfigurations

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Lists the metrics configurations for a bucket. The fluent builder allows setting the bucket name, continuation token, and expected bucket owner. On success, returns a ListBucketMetricsConfigurationsOutput with fields indicating truncation status and continuation token.

```APIDOC
## ListBucketMetricsConfigurations

### Description
Lists the metrics configurations for a bucket. Use the fluent builder to configure the request parameters.

### Method
N/A (SDK operation)

### Parameters
#### Fluent Builder Methods
- **bucket** (`impl Into<String>`) / **set_bucket** (`Option<String>`) - Required - The name of the bucket containing the metrics configurations to retrieve.
- **continuation_token** (`impl Into<String>`) / **set_continuation_token** (`Option<String>`) - Optional - The marker that is used to continue a metrics configuration listing that has been truncated. Use the `NextContinuationToken` from a previously truncated list response to continue the listing. The continuation token is an opaque value that Amazon S3 understands.
- **expected_bucket_owner** (`impl Into<String>`) / **set_expected_bucket_owner** (`Option<String>`) - Optional - The account ID of the expected bucket owner. If the account ID that you provide does not match the actual owner of the bucket, the request fails with the HTTP status code `403 Forbidden` (access denied).

### Response
#### Success Response
- **is_truncated** (`Option<bool>`) - Indicates whether the returned list of metrics configurations is complete. A value of true indicates that the list is not complete and the NextContinuationToken will be provided for a subsequent request.
- **continuation_token** (`Option<String>`) - The marker that is used as a starting point for this metrics configuration list response. This value is present if it was sent in the request.

```

--------------------------------

### sse_customer_algorithm

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Specifies the algorithm to use when encrypting the object (for example, AES256). This functionality is not supported for directory buckets.

```APIDOC
## sse_customer_algorithm

### Description
Specifies the algorithm to use when encrypting the object (for example, AES256). This functionality is not supported for directory buckets.

### Method
sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

### Parameters
- **value** (String) - Required: false - The encryption algorithm to use.

### Notes
- Not supported for directory buckets.
```

--------------------------------

### create

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.AwsS3Fs.html

Creates a file at the given path for writing. If the file already exists, its content will be overwritten.

```APIDOC
## create

### Description
Creates a file at the given path for writing. If the file already exists, its content will be overwritten.

### Method
fn create(&mut self, _path: &Path, _metadata: &Metadata) -> RemoteResult<WriteStream>

### Parameters
- `_path` (&Path) - The path to the file.
- `_metadata` (&Metadata) - The metadata for the file.

### Returns
- `RemoteResult<WriteStream>` - A write stream for writing data.

```

--------------------------------

### S3Client::sse_customer_algorithm

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Specifies the algorithm to use when encrypting the object (for example, AES256). Not supported for directory buckets.

```APIDOC
## sse_customer_algorithm(impl Into<String>)

### Description
Specifies the algorithm to use when encrypting the object (for example, AES256). Not supported for directory buckets.

### Method
sse_customer_algorithm(impl Into<String>)

### Parameters
- **value** (impl Into<String>) - Required - The encryption algorithm name.

### Returns
- **Self** - The builder instance for chaining.

### Notes
- Required: false
- Not supported for directory buckets.
```

--------------------------------

### key(impl Into<String>) / set_key(Option<String>)

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Sets the key of the object to get. This parameter is required.

```APIDOC
## key(impl Into<String>) / set_key(Option<String>)

### Description
Sets the key of the object to get. This parameter is required.

### Method
Builder method on `GetObjectFluentBuilder`

### Parameters
- `key` (String) - Required - The key of the object to get.
```

--------------------------------

### list_bucket_metrics_configurations

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Constructs a fluent builder for the ListBucketMetricsConfigurations operation. On success, returns a ListBucketMetricsConfigurationsOutput with fields for continuation token, metrics configuration list, truncation status, and next continuation token. On failure, returns an SdkError with ListBucketMetricsConfigurationsError.

```APIDOC
## list_bucket_metrics_configurations

### Description
Constructs a fluent builder for the `ListBucketMetricsConfigurations` operation. This method is part of the `S3Client` struct.

### Method
`pub fn list_bucket_metrics_configurations(&self) -> ListBucketMetricsConfigurationsFluentBuilder`

### Parameters
None.

### Returns
- On success: `ListBucketMetricsConfigurationsOutput` with fields:
  - `continuation_token` (Option<String>) - If sent in the request, the marker that is used as a starting point for this metrics configuration list response.
  - `metrics_configuration_list` (Option<Vec<MetricsConfiguration>>) - The list of metrics configurations for a bucket.
  - `is_truncated` (Option<bool>) - Indicates whether the returned list of metrics configurations is complete. A value of true indicates the list is not complete and the NextContinuationToken is provided for a subsequent request.
  - `next_continuation_token` (Option<String>) - The marker used to continue this metrics configuration listing. Use the NextContinuationToken from this response to continue the listing in a subsequent request.
- On failure: `SdkError<ListBucketMetricsConfigurationsError>`
```

--------------------------------

### sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Specifies the algorithm to use when encrypting the object (for example, AES256).

```APIDOC
## sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

### Description
Specifies the algorithm to use when encrypting the object (for example, `AES256`).

### Method
`sse_customer_algorithm(impl Into<String>)` / `set_sse_customer_algorithm(Option<String>)`

### Parameters
- **sse_customer_algorithm** (String) - Required: false - The server-side encryption algorithm to use.
```

--------------------------------

### sse_customer_algorithm / set_sse_customer_algorithm

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Sets the algorithm to use when encrypting the object (for example, AES256). Not supported when the destination bucket is a directory bucket.

```APIDOC
## sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

### Description
Specifies the algorithm to use when encrypting the object (for example, AES256). This functionality is not supported when the destination bucket is a directory bucket.

### Parameters
- **value** (String) - Required - The encryption algorithm to use.

### Returns
- The builder with the SSE customer algorithm set.
```

--------------------------------

### AwsS3Fs builder methods: new, region, endpoint, profile, new_path_style, access_key, secret_access_key, security_token, session_token

Source: https://docs.rs/remotefs-aws-s3/0.4.3/src/remotefs_aws_s3/client.rs.html

Builder-style methods for configuring AwsS3Fs. new initializes with a bucket and runtime; region sets the region; endpoint sets a custom endpoint (e.g., MinIO); profile sets the AWS profile (default "default"); new_path_style enables new path style (required for MinIO); access_key, secret_access_key, security_token, and session_token set credentials, falling back to environment variables if unset.

```rust
impl AwsS3Fs {
    /// Initialize a new `AwsS3Fs`
    pub fn new<S: AsRef<str>>(bucket: S, runtime: &Arc<Runtime>) -> Self {
        Self {
            client: None,
            runtime: runtime.clone(),
            wrkdir: PathBuf::from("/"),
            bucket_name: bucket.as_ref().to_string(),
            region: None,
            endpoint: None,
            profile: None,
            access_key: None,
            secret_key: None,
            security_token: None,
            session_token: None,
            new_path_style: false,
        }
    }

    /// Specify region to connect to
    pub fn region<S: AsRef<str>>(mut self, region: S) -> Self {
        self.region = Some(region.as_ref().to_string());
        self
    }

    /// Specify custom endpoint
    /// This should be used when trying to connect to `minio` or other API compatible endpoints.
    pub fn endpoint<S: AsRef<str>>(mut self, endpoint: S) -> Self {
        self.endpoint = Some(endpoint.as_ref().to_string());
        self
    }

    /// Set aws profile. If unset, "default" will be used
    pub fn profile<S: AsRef<str>>(mut self, profile: S) -> Self {
        self.profile = Some(profile.as_ref().to_string());
        self
    }

    /// Set whether to use new path style. Required for backends such as MinIO (Default: False)
    pub fn new_path_style(mut self, new_path_style: bool) -> Self {
        self.new_path_style = new_path_style;
        self
    }

    /// Specify access key for aws connection.
    /// If unset, will be read from environment variable `AWS_ACCESS_KEY_ID`
    pub fn access_key<S: AsRef<str>>(mut self, key: S) -> Self {
        self.access_key = Some(key.as_ref().to_string());
        self
    }

    /// Specify secret access key for aws connection.
    /// If unset, will be read from environment variable `AWS_SECRET_ACCESS_KEY`
    pub fn secret_access_key<S: AsRef<str>>(mut self, key: S) -> Self {
        self.secret_key = Some(key.as_ref().to_string());
        self
    }

    /// Specify security token for aws connection.
    /// If unset, will be read from environment variable `AWS_SECURITY_TOKEN`
    pub fn security_token<S: AsRef<str>>(mut self, key: S) -> Self {
        self.security_token = Some(key.as_ref().to_string());
        self
    }

    /// Specify session token for aws connection.
    /// If unset, will be read from environment variable `AWS_SESSION_TOKEN`
    pub fn session_token<S: AsRef<str>>(mut self, key: S) -> Self {
        self.session_token = Some(key.as_ref().to_string());
        self
    }
}
```

--------------------------------

### sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

Source: https://docs.rs/remotefs-aws-s3/0.4.3/remotefs_aws_s3/client/struct.S3Client.html

Specifies the algorithm to use when encrypting the object (for example, AES256). This parameter is optional and not supported for directory buckets.

```APIDOC
## sse_customer_algorithm(impl Into<String>) / set_sse_customer_algorithm(Option<String>)

### Description
Specifies the algorithm to use when encrypting the object (for example, AES256). This parameter is optional and not supported for directory buckets.

### Parameters
- **sse_customer_algorithm** (impl Into<String>) - Optional - The algorithm to use when encrypting the object.
- **set_sse_customer_algorithm** (Option<String>) - Optional - Sets the SSE customer algorithm using an Option.

### Notes
- This functionality is not supported for directory buckets.
```
Tokens:
Raw