### Install HDFS Agent with Simple Authentication Source: https://cloud.google.com/storage-transfer/docs/create-transfers/agent-based/hdfs?hl=es-419 This command installs an HDFS agent using simple authentication. Replace placeholders like PROJECT_ID, CREDS_FILE, AGENT_POOL_NAME, and USERNAME with your specific values. ```bash sudo docker run -d --ulimit memlock=64000000 --rm \ --network=host \ -v /:/transfer_root \ gcr.io/cloud-ingest/tsop-agent:latest \ --enable-mount-directory \ --project-id=${PROJECT_ID} \ --hostname=$(hostname) \ --creds-file="${CREDS_FILE}" \ --agent-pool="${AGENT_POOL_NAME}" \ --hdfs-namenode-uri=cluster-namenode \ --hdfs-username="${USERNAME}" ``` -------------------------------- ### Install HDFS Agent with Kerberos Authentication Source: https://cloud.google.com/storage-transfer/docs/create-transfers/agent-based/hdfs?hl=es-419 Use this command to install an HDFS agent when Kerberos authentication is enabled. Ensure you replace placeholder values with your specific Kerberos configuration. ```bash sudo docker run -d --ulimit memlock=64000000 --rm \ --network=host \ -v /:/transfer_root \ gcr.io/cloud-ingest/tsop-agent:latest \ --enable-mount-directory \ --project-id=${PROJECT_ID} \ --hostname=$(hostname) \ --creds-file="${CREDS_FILE}" \ --agent-pool="${AGENT_POOL_NAME}" \ --hdfs-namenode-uri=cluster-namenode \ --kerberos-user-principal=user \ --kerberos-keytab-file=/path/to/folder.keytab ``` -------------------------------- ### Create and run a POSIX to POSIX transfer job (Node.js) Source: https://cloud.google.com/storage-transfer/docs/file-to-file?hl=id This example demonstrates creating a transfer job from a POSIX filesystem to a POSIX sink, utilizing GCS for intermediate data. It uses async/await for handling the asynchronous operations. Ensure the client is initialized with appropriate project and agent pool details. ```javascript // Imports the Google Cloud client library const { StorageTransferServiceClient, } = require('@google-cloud/storage-transfer'); /** * TODO(developer): Uncomment the following lines before running the sample. */ // Your project id // const projectId = 'my-project' // The agent pool associated with the POSIX data source. Defaults to the default agent // const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default' // The agent pool associated with the POSIX data sink. Defaults to the default agent // const sinkAgentPoolName = 'projects/my-project/agentPools/transfer_service_default' // The root directory path on the source filesystem // const rootDirectory = '/directory/to/transfer/source' // The root directory path on the sink filesystem // const destinationDirectory = '/directory/to/transfer/sink' // The ID of the GCS bucket for intermediate storage // const bucketName = 'my-intermediate-bucket' // Creates a client const client = new StorageTransferServiceClient(); /** * Creates a request to transfer from the local file system to the sink bucket */ async function transferDirectory() { const createRequest = { transferJob: { projectId, transferSpec: { sourceAgentPoolName, sinkAgentPoolName, posixDataSource: { rootDirectory, }, posixDataSink: { rootDirectory: destinationDirectory, }, gcsIntermediateDataLocation: { bucketName, }, }, status: 'ENABLED', }, }; // Runs the request and creates the job const [transferJob] = await client.createTransferJob(createRequest); const runRequest = { jobName: transferJob.name, projectId: projectId, }; await client.runTransferJob(runRequest); console.log( `Created and ran a transfer job from '${rootDirectory}' to '${destinationDirectory}' with name ${transferJob.name}` ); } transferDirectory(); ``` -------------------------------- ### Install HDFS Agent without Authentication Source: https://cloud.google.com/storage-transfer/docs/create-transfers/agent-based/hdfs?hl=es-419 Use this command to install an HDFS agent when no specific authentication method is required. This is suitable for environments where access is managed at the network level. ```bash sudo docker run -d --ulimit memlock=64000000 --rm \ --network=host \ -v /:/transfer_root \ gcr.io/cloud-ingest/tsop-agent:latest \ --enable-mount-directory \ --project-id=${PROJECT_ID} \ --hostname=$(hostname) \ --creds-file="${CREDS_FILE}" \ --agent-pool="${AGENT_POOL_NAME}" \ --hdfs-namenode-uri=cluster-namenode \ ``` -------------------------------- ### Create a Transfer Job (Node.js) Source: https://cloud.google.com/storage-transfer/docs/libraries This Node.js snippet shows how to initialize the Storage Transfer Service client. It is a prerequisite for creating and managing transfer jobs. Ensure your environment is authenticated, for example, by configuring Application Default Credentials. ```javascript // Imports the Google Cloud client library const { StorageTransferServiceClient, } = require('@google-cloud/storage-transfer'); /** * TODO(developer): Uncomment the following lines before running the sample. */ // Your project id // const projectId = 'my-project' // The ID of the GCS bucket to transfer data from // const gcsSourceBucket = 'my-source-bucket' // The ID of the GCS bucket to transfer data to // const gcsSinkBucket = 'my-sink-bucket' // Creates a client const client = new StorageTransferServiceClient(); /** * Creates a one-time transfer job. */ ``` -------------------------------- ### HttpData List URL Example Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 Specifies a list of objects on the web to be transferred over HTTP. The file must start with 'TsvHttpData-1.0' and subsequent lines specify object information in tab-delimited format: HTTP URL, Length, and MD5. ```text TsvHttpData-1.0 https://example.com/object1.txt 1024 Xo870g63Xq9x0123456789== https://example.com/object2.txt 2048 Yp987f74Yq8y0123456789== ``` -------------------------------- ### Duration Format Example Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/ObjectConditions Demonstrates the required format for specifying time durations in seconds, including fractional digits, for time-based conditions like minTimeElapsedSinceLastModification and maxTimeElapsedSinceLastModification. ```string "3.5s" ``` -------------------------------- ### Create a transfer job using a manifest (Node.js) Source: https://cloud.google.com/storage-transfer/docs/manifest?hl=es This Node.js snippet demonstrates how to initiate a transfer from a local filesystem to a GCS bucket using a manifest file. Ensure you have the `@google-cloud/storage-transfer` library installed and authentication configured. ```javascript // Imports the Google Cloud client library const { StorageTransferServiceClient, } = require('@google-cloud/storage-transfer'); /** * TODO(developer): Uncomment the following lines before running the sample. */ // Your project id // const projectId = 'my-project' // The agent pool associated with the POSIX data source. Defaults to the default agent // const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default' // The root directory path on the source filesystem // const rootDirectory = '/directory/to/transfer/source' // The ID of the GCS bucket to transfer data to // const gcsSinkBucket = 'my-sink-bucket' // Transfer manifest location. Must be a `gs:` URL // const manifestLocation = 'gs://my-bucket/sample_manifest.csv' // Creates a client const client = new StorageTransferServiceClient(); /** * Creates a request to transfer from the local file system to the sink bucket */ async function transferViaManifest() { const createRequest = { transferJob: { projectId, ``` -------------------------------- ### Transfer Schedule Configuration Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 Configure the scheduling parameters for a transfer job, including start and end dates, start time, and repeat interval. ```APIDOC ## Transfer Schedule Configuration ### Description Configure the scheduling parameters for a transfer job, including start and end dates, start time, and repeat interval. ### Fields - **`schedule_start_date`** (`Date`) - Required. The start date of a transfer. Date boundaries are determined relative to UTC time. If `schedule_start_date` and `start_time_of_day` are in the past relative to the job's creation time, the transfer starts the day after you schedule the transfer request. - **`schedule_end_date`** (`Date`) - Optional. The last day a transfer runs. Date boundaries are determined relative to UTC time. - **`start_time_of_day`** (`TimeOfDay`) - Optional. The time in UTC that a transfer job is scheduled to run. Transfers may start later than this time. - **`end_time_of_day`** (`TimeOfDay`) - Optional. The time in UTC that no further transfer operations are scheduled. Combined with `schedule_end_date`, `end_time_of_day` specifies the end date and time for starting new transfer operations. - **`repeat_interval`** (`Duration`) - Optional. Interval between the start of each scheduled TransferOperation. If unspecified, the default value is 24 hours. This value may not be less than 1 hour. ``` -------------------------------- ### ListTransferJobsRequest Filter Example Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 A required list of query parameters to filter transfer jobs. Parameters like projectId, jobNames, jobStatuses, dataBackend, sourceBucket, and sinkBucket can be specified. Provide the query parameters without spaces or line breaks. ```json { "projectId":"my_project_id", "jobNames":["jobid1","jobid2"], "jobStatuses":["status1","status2"], "dataBackend":"QUERY_REPLICATION_CONFIGS", "sourceBucket":"source-bucket-name", "sinkBucket":"sink-bucket-name" } ``` -------------------------------- ### Create and Run Transfer Job (PHP) Source: https://cloud.google.com/storage-transfer/docs/libraries Creates and runs a transfer job between two GCS buckets using the PHP client library. Ensure you have the client library installed and authenticated. ```php use Google\Cloud\StorageTransfer\V1\Client\StorageTransferServiceClient; use Google\Cloud\StorageTransfer\V1\CreateTransferJobRequest; use Google\Cloud\StorageTransfer\V1\GcsData; use Google\Cloud\StorageTransfer\V1\RunTransferJobRequest; use Google\Cloud\StorageTransfer\V1\TransferJob; use Google\Cloud\StorageTransfer\V1\TransferJob\Status; use Google\Cloud\StorageTransfer\V1\TransferSpec; /** * Creates and runs a transfer job between two GCS buckets * * @param string $projectId Your Google Cloud project ID. * @param string $sourceGcsBucketName The name of the GCS bucket to transfer objects from. * @param string $sinkGcsBucketName The name of the GCS bucket to transfer objects to. */ function quickstart( string $projectId, string $sourceGcsBucketName, string $sinkGcsBucketName ): void { // $project = 'my-project-id'; // $sourceGcsBucketName = 'my-source-bucket'; // $sinkGcsBucketName = 'my-sink-bucket'; $transferJob = new TransferJob([ 'project_id' => $projectId, 'transfer_spec' => new TransferSpec([ 'gcs_data_sink' => new GcsData(['bucket_name' => $sinkGcsBucketName]), 'gcs_data_source' => new GcsData(['bucket_name' => $sourceGcsBucketName]) ]), 'status' => Status::ENABLED ]); $client = new StorageTransferServiceClient(); $createRequest = (new CreateTransferJobRequest()) ->setTransferJob($transferJob); $response = $client->createTransferJob($createRequest); $runRequest = (new RunTransferJobRequest()) ->setJobName($response->getName()) ->setProjectId($projectId); $client->runTransferJob($runRequest); printf('Created and ran transfer job from %s to %s with name %s ' . PHP_EOL, $sourceGcsBucketName, $sinkGcsBucketName, $response->getName()); } ``` -------------------------------- ### Create a Transfer Job Source: https://cloud.google.com/storage-transfer/docs/create-transfers Use the gcloud transfer jobs create command to initiate a new transfer job. Specify source and destination paths. The job starts immediately unless a schedule or --do-not-run option is specified. ```bash gcloud transfer jobs create \ SOURCE DESTINATION ``` -------------------------------- ### Run Transfer Job Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs Starts a new operation for a specified transfer job. This is the primary method for initiating data transfers. ```APIDOC ## POST /v1/transferJobs/{transferJobId}:run ### Description Starts a new operation for the specified transfer job. ### Method POST ### Endpoint /v1/transferJobs/{transferJobId}:run ### Parameters #### Path Parameters - **transferJobId** (string) - Required - The ID of the transfer job to run. ### Request Body This operation does not require a request body. ### Response #### Success Response (200) This operation returns an empty response body on success. #### Response Example (No response body) ``` -------------------------------- ### Create a Cloud Storage Transfer Job Source: https://cloud.google.com/storage-transfer/docs/create-transfers This Python snippet shows how to construct and send a request to create a transfer job. It includes setting the source and destination buckets, a description, an enabled status, a start date, and transfer options such as deleting source objects after a successful transfer. It also configures object conditions like minimum elapsed time since last modification. ```Python # A useful description for your transfer job # description = 'My transfer job' # Google Cloud Storage source bucket name # source_bucket = 'my-gcs-source-bucket' # Google Cloud Storage destination bucket name # sink_bucket = 'my-gcs-destination-bucket' transfer_job_request = storage_transfer.CreateTransferJobRequest( { "transfer_job": { "project_id": project_id, "description": description, "status": storage_transfer.TransferJob.Status.ENABLED, "schedule": { "schedule_start_date": { "day": start_date.day, "month": start_date.month, "year": start_date.year, } }, "transfer_spec": { "gcs_data_source": { "bucket_name": source_bucket, }, "gcs_data_sink": { "bucket_name": sink_bucket, }, "object_conditions": { "min_time_elapsed_since_last_modification": Duration( seconds=2592000 # 30 days ) }, "transfer_options": { "delete_objects_from_source_after_transfer": True }, }, } } ) result = client.create_transfer_job(transfer_job_request) print(f"Created transferJob: {result.name}") ``` -------------------------------- ### Create One-Time Transfer Job (Python) Source: https://cloud.google.com/storage-transfer/docs/libraries Creates a one-time transfer job between two GCS buckets using the Python client library. Ensure you have the client library installed and authenticated. ```python from google.cloud import storage_transfer def create_one_time_transfer( project_id: str = "my_project_id", source_bucket: str = "my_source_bucket", sink_bucket: str = "my_sink_bucket", ): """Creates a one-time transfer job.""" client = storage_transfer.StorageTransferServiceClient() # The ID of the Google Cloud Platform Project that owns the job # project_id = 'my-project-id' # Google Cloud Storage source bucket name # source_bucket = 'my-gcs-source-bucket' # Google Cloud Storage destination bucket name # sink_bucket = 'my-gcs-destination-bucket' transfer_job_request = storage_transfer.CreateTransferJobRequest( { "transfer_job": { "project_id": project_id, "status": storage_transfer.TransferJob.Status.ENABLED, "transfer_spec": { "gcs_data_source": { "bucket_name": source_bucket, }, "gcs_data_sink": { "bucket_name": sink_bucket, }, }, } } ) transfer_job = client.create_transfer_job(transfer_job_request) client.run_transfer_job({"job_name": transfer_job.name, "project_id": project_id}) print(f"Created and ran transfer job: {transfer_job.name}") ``` -------------------------------- ### Create and Run Transfer Job (Node.js) Source: https://cloud.google.com/storage-transfer/docs/libraries Creates and runs a transfer job between two GCS buckets using the Node.js client library. Ensure you have the client library installed and authenticated. ```javascript async function quickstart() { // Creates a request to transfer from the source bucket to // the sink bucket const createRequest = { transferJob: { projectId: projectId, transferSpec: { gcsDataSource: {bucketName: gcsSourceBucket}, gcsDataSink: {bucketName: gcsSinkBucket}, }, status: 'ENABLED', }, }; // Runs the request and creates the job const [transferJob] = await client.createTransferJob(createRequest); const runRequest = { jobName: transferJob.name, projectId: projectId, }; await client.runTransferJob(runRequest); console.log( `Created and ran a transfer job from ${gcsSourceBucket} to ${gcsSinkBucket} with name ${transferJob.name}` ); } quickstart(); ``` -------------------------------- ### Python: Transfer Directory from POSIX to GCS Source: https://cloud.google.com/storage-transfer/docs/file-to-file?hl=fr This Python snippet sets up and initiates a transfer job from a POSIX file system to a Google Cloud Storage bucket. Ensure the Google Cloud client library for Storage Transfer Service is installed and configured. ```python from google.cloud import storage_transfer def transfer_between_posix( project_id: str, description: str, source_agent_pool_name: str, sink_agent_pool_name: str, root_directory: str, destination_directory: str, intermediate_bucket: str, ): """Creates a transfer between POSIX file systems.""" client = storage_transfer.StorageTransferServiceClient() # The ID of the Google Cloud Platform Project that owns the job # project_id = 'my-project-id' # A useful description for your transfer job # description = 'My transfer job' # The agent pool associated with the POSIX data source. # Defaults to 'projects/{project_id}/agentPools/transfer_service_default' # source_agent_pool_name = 'projects/my-project/agentPools/my-agent' # The agent pool associated with the POSIX data sink. # Defaults to 'projects/{project_id}/agentPools/transfer_service_default' # sink_agent_pool_name = 'projects/my-project/agentPools/my-agent' # The root directory path on the source filesystem # root_directory = '/directory/to/transfer/source' # The root directory path on the destination filesystem # destination_directory = '/directory/to/transfer/sink' # The Google Cloud Storage bucket for intermediate storage # intermediate_bucket = 'my-intermediate-bucket' transfer_job_request = storage_transfer.CreateTransferJobRequest( { "transfer_job": { "project_id": project_id, "description": description, "status": storage_transfer.TransferJob.Status.ENABLED, "transfer_spec": { "source_agent_pool_name": source_agent_pool_name, "sink_agent_pool_name": sink_agent_pool_name, "posix_data_source": { "root_directory": root_directory, }, "posix_data_sink": { "root_directory": destination_directory, }, "gcs_intermediate_data_location": { "bucket_name": intermediate_bucket }, }, } } ) result = client.create_transfer_job(transfer_job_request) print(f"Created transferJob: {result.name}") ``` -------------------------------- ### Crear y ejecutar un trabajo de transferencia con C# Source: https://cloud.google.com/storage-transfer/docs/libraries?hl=es Este snippet demuestra cómo crear y ejecutar un trabajo de transferencia de un bucket de Google Cloud Storage a otro utilizando la biblioteca cliente de C# de Storage Transfer Service. Requiere la configuración de credenciales predeterminadas de la aplicación. ```csharp using Google.Cloud.StorageTransfer.V1; using System; public class QuickStartSample { /// /// Creates an one-time transfer job from a Google Cloud storage bucket to another bucket. /// /// The ID of the Google Cloud project. /// The GCS bucket to transfer data from. /// The GCS bucket to transfer data to. public TransferJob QuickStart( string projectId = "my-project-id", string sourceBucket = "my-source-bucket", string sinkBucket = "my-sink-bucket") { TransferJob transferJob = new TransferJob { ProjectId = projectId, TransferSpec = new TransferSpec { GcsDataSink = new GcsData { BucketName = sinkBucket }, GcsDataSource = new GcsData { BucketName = sourceBucket } }, Status = TransferJob.Types.Status.Enabled }; StorageTransferServiceClient client = StorageTransferServiceClient.Create(); TransferJob response = client.CreateTransferJob(new CreateTransferJobRequest { TransferJob = transferJob }); client.RunTransferJob(new RunTransferJobRequest { JobName = response.Name, ProjectId = projectId }); Console.WriteLine($"Created and ran transfer job from {sourceBucket} to {sinkBucket} with name {response.Name}"); return response; } } ``` -------------------------------- ### Create and Run Transfer Job from POSIX Filesystem (Node.js) Source: https://cloud.google.com/storage-transfer/docs/create-transfers/agent-based/file-system-to-cloud-storage?hl=id This Node.js sample demonstrates how to create and run a transfer job from a POSIX filesystem to a GCS bucket. It uses the Google Cloud Storage Transfer Service client library for Node.js. ```javascript // Imports the Google Cloud client library const { StorageTransferServiceClient, } = require('@google-cloud/storage-transfer'); /** * TODO(developer): Uncomment the following lines before running the sample. */ // Your project id // const projectId = 'my-project' // The agent pool associated with the POSIX data source. Defaults to the default agent // const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default' // The root directory path on the source filesystem // const rootDirectory = '/directory/to/transfer/source' // The ID of the GCS bucket to transfer data to // const gcsSinkBucket = 'my-sink-bucket' // Creates a client const client = new StorageTransferServiceClient(); /** * Creates a request to transfer from the local file system to the sink bucket */ async function transferDirectory() { const createRequest = { transferJob: { projectId, transferSpec: { sourceAgentPoolName, posixDataSource: { rootDirectory, }, gcsDataSink: {bucketName: gcsSinkBucket}, }, status: 'ENABLED', }, }; // Runs the request and creates the job const [transferJob] = await client.createTransferJob(createRequest); const runRequest = { jobName: transferJob.name, projectId: projectId, }; await client.runTransferJob(runRequest); console.log( `Created and ran a transfer job from '${rootDirectory}' to '${gcsSinkBucket}' with name ${transferJob.name}` ); } transferDirectory(); ``` -------------------------------- ### get Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations Retrieves the latest status of a long-running transfer operation. ```APIDOC ## get ### Description Gets the latest state of a long-running operation. ### Method GET ### Endpoint /v1/transferOperations/{operationName} ### Parameters #### Path Parameters - **operationName** (string) - Required - The name of the operation to retrieve. ### Response #### Success Response (200) - **operation** (object) - The current state of the transfer operation. ``` -------------------------------- ### Create and Run a Transfer Job (Java) Source: https://cloud.google.com/storage-transfer/docs/libraries This Java sample demonstrates creating and running a transfer job between two GCS buckets. It utilizes the Storage Transfer Service client library. Ensure you have set up Application Default Credentials for authentication. ```java import com.google.storagetransfer.v1.proto.StorageTransferServiceClient; import com.google.storagetransfer.v1.proto.TransferProto.CreateTransferJobRequest; import com.google.storagetransfer.v1.proto.TransferProto.RunTransferJobRequest; import com.google.storagetransfer.v1.proto.TransferTypes.GcsData; import com.google.storagetransfer.v1.proto.TransferTypes.TransferJob; import com.google.storagetransfer.v1.proto.TransferTypes.TransferSpec; public class QuickstartSample { /** Quickstart sample using transfer service to transfer from one GCS bucket to another. */ public static void main(String[] args) throws Exception { // TODO(developer): Replace these variables before running the sample. // Your Google Cloud Project ID String projectId = "your-project-id"; // The name of the source GCS bucket to transfer objects from String gcsSourceBucket = "your-source-gcs-source-bucket"; // The name of the GCS bucket to transfer objects to String gcsSinkBucket = "your-sink-gcs-bucket"; quickStartSample(projectId, gcsSourceBucket, gcsSinkBucket); } public static void quickStartSample( String projectId, String gcsSourceBucket, String gcsSinkBucket) throws Exception { // Initialize client that will be used to send requests. This client only needs to be created // once, and can be reused for multiple requests. After completing all of your requests, call // the "close" method on the client to safely clean up any remaining background resources, // or use "try-with-close" statement to do this automatically. try (StorageTransferServiceClient storageTransfer = StorageTransferServiceClient.create()) { TransferJob transferJob = TransferJob.newBuilder() .setProjectId(projectId) .setTransferSpec( TransferSpec.newBuilder() .setGcsDataSource(GcsData.newBuilder().setBucketName(gcsSourceBucket)) .setGcsDataSink(GcsData.newBuilder().setBucketName(gcsSinkBucket))) .setStatus(TransferJob.Status.ENABLED) .build(); TransferJob response = storageTransfer.createTransferJob( CreateTransferJobRequest.newBuilder().setTransferJob(transferJob).build()); storageTransfer .runTransferJobAsync( RunTransferJobRequest.newBuilder() .setProjectId(projectId) .setJobName(response.getName()) .build()) .get(); System.out.println( "Created and ran transfer job between two GCS buckets with name " + response.getName()); } } } ``` -------------------------------- ### Crear y ejecutar un trabajo de transferencia con Go Source: https://cloud.google.com/storage-transfer/docs/libraries?hl=es Este snippet muestra cómo crear y ejecutar un trabajo de transferencia entre dos buckets de GCS utilizando la biblioteca cliente de Go de Storage Transfer Service. Asegúrate de que el contexto y el cliente se manejen correctamente. ```go import ( "context" "fmt" "io" storagetransfer "cloud.google.com/go/storagetransfer/apiv1" "cloud.google.com/go/storagetransfer/apiv1/storagetransferpb" ) // quickstart creates and runs a transfer job between two GCS buckets. func quickstart(w io.Writer, projectID string, sourceGCSBucket string, sinkGCSBucket string) (*storagetransferpb.TransferJob, error) { // Your Google Cloud Project ID // projectID := "my-project-id" // The name of the GCS bucket to transfer data from // sourceGCSBucket := "my-source-bucket" // The name of the GCS bucket to transfer data to sinkGCSBucket := "my-sink-bucket" ctx := context.Background() client, err := storagetransfer.NewClient(ctx) if err != nil { return nil, fmt.Errorf("storagetransfer.NewClient: %w", err) } defer client.Close() req := &storagetransferpb.CreateTransferJobRequest{ ``` -------------------------------- ### Get Transfer Job Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs Retrieves the details of a specific transfer job. ```APIDOC ## GET /v1/transferJobs/{jobName} ### Description Gets the details of a specific transfer job. ### Method GET ### Endpoint /v1/transferJobs/{jobName} ### Parameters #### Path Parameters - **jobName** (string) - Required. The name of the transfer job to retrieve. ### Response #### Success Response (200) - **name** (string) - The unique name of the transfer job. - **status** (string) - The status of the transfer job. #### Response Example ```json { "name": "transferJobs/12345", "status": "ENABLED" } ``` ``` -------------------------------- ### Listar trabajos de transferencia con C++ Source: https://cloud.google.com/storage-transfer/docs/libraries?hl=es Este snippet muestra cómo listar trabajos de transferencia existentes para un proyecto específico usando la biblioteca cliente de C++ de Storage Transfer Service. Asegúrate de tener configuradas las credenciales predeterminadas de la aplicación. ```cpp #include "google/cloud/storagetransfer/v1/storage_transfer_client.h" #include #include int main(int argc, char* argv[]) try { if (argc != 2) { std::cerr << "Usage: " << argv[0] << " project-id\n"; return 1; } namespace storagetransfer = ::google::cloud::storagetransfer_v1; auto client = storagetransfer::StorageTransferServiceClient( storagetransfer::MakeStorageTransferServiceConnection()); ::google::storagetransfer::v1::ListTransferJobsRequest request; request.set_filter(R"({\"projectId": ")" + std::string{argv[1]} + ""}"); for (auto r : client.ListTransferJobs(request)) { if (!r) throw std::move(r).status(); std::cout << r->DebugString() << "\n"; } return 0; } catch (google::cloud::Status const& status) { std::cerr << "google::cloud::Status thrown: " << status << "\n"; return 1; } ``` -------------------------------- ### GetTransferJob Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 Gets a transfer job. This retrieves the details of a specific transfer job. ```APIDOC ## GetTransferJob ### Description Gets a transfer job. ### Method `rpc GetTransferJob(GetTransferJobRequest) returns (TransferJob)` ### Authorization Requires the following OAuth scope: `https://www.googleapis.com/auth/cloud-platform` ``` -------------------------------- ### GetAgentPool Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 Gets an agent pool. This retrieves the details of a specific agent pool. ```APIDOC ## GetAgentPool ### Description Gets an agent pool. ### Method `rpc GetAgentPool(GetAgentPoolRequest) returns (AgentPool)` ### Authorization Requires the following OAuth scope: `https://www.googleapis.com/auth/cloud-platform` ``` -------------------------------- ### Schedule Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs Defines the scheduling parameters for a transfer job, including start and end dates, times, and repeat intervals. ```APIDOC ## Schedule Transfers can be scheduled to recur or to run just once. ### JSON representation ```json { "scheduleStartDate": { "object": "Date" }, "scheduleEndDate": { "object": "Date" }, "startTimeOfDay": { "object": "TimeOfDay" }, "endTimeOfDay": { "object": "TimeOfDay" }, "repeatInterval": "string" } ``` ### Fields * `scheduleStartDate` (Date) - Required. The start date of a transfer. Date boundaries are determined relative to UTC time. If `scheduleStartDate` and `startTimeOfDay` are in the past relative to the job's creation time, the transfer starts the day after you schedule the transfer request. Note: When starting jobs at or near midnight UTC it is possible that a job starts later than expected. For example, if you send an outbound request on June 1 one millisecond prior to midnight UTC and the Storage Transfer Service server receives the request on June 2, then it creates a TransferJob with `scheduleStartDate` set to June 2 and a `startTimeOfDay` set to midnight UTC. The first scheduled `TransferOperation` takes place on June 3 at midnight UTC. * `scheduleEndDate` (Date) - The last day a transfer runs. Date boundaries are determined relative to UTC time. A job runs once per 24 hours within the following guidelines: * If `scheduleEndDate` and `scheduleStartDate` are the same and in the future relative to UTC, the transfer is executed only one time. * If `scheduleEndDate` is later than `scheduleStartDate` and `scheduleEndDate` is in the future relative to UTC, the job runs each day at `startTimeOfDay` through `scheduleEndDate`. * `startTimeOfDay` (TimeOfDay) - The time in UTC that a transfer job is scheduled to run. Transfers may start later than this time. If `startTimeOfDay` is not specified: * One-time transfers run immediately. * Recurring transfers run immediately, and each day at midnight UTC, through `scheduleEndDate`. If `startTimeOfDay` is specified: * One-time transfers run at the specified time. * Recurring transfers run at the specified time each day, through `scheduleEndDate`. * `endTimeOfDay` (TimeOfDay) - The time in UTC that no further transfer operations are scheduled. Combined with `scheduleEndDate`, `endTimeOfDay` specifies the end date and time for starting new transfer operations. This field must be greater than or equal to the timestamp corresponding to the combination of `scheduleStartDate` and `startTimeOfDay`, and is subject to the following: * If `endTimeOfDay` is not set and `scheduleEndDate` is set, then a default value of `23:59:59` is used for `endTimeOfDay`. * If `endTimeOfDay` is set and `scheduleEndDate` is not set, then `INVALID_ARGUMENT` is returned. * `repeatInterval` (string (Duration format)) - Interval between the start of each scheduled TransferOperation. If unspecified, the default value is 24 hours. This value may not be less than 1 hour. A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s". ``` -------------------------------- ### Create a transfer job using a manifest file (Node.js) Source: https://cloud.google.com/storage-transfer/docs/manifest?hl=pt-BR This Node.js code snippet demonstrates how to initiate a data transfer from a POSIX filesystem to a GCS bucket using a manifest file. It utilizes the Google Cloud Storage Transfer Service client library. Remember to uncomment and set the placeholder variables. ```javascript // Imports the Google Cloud client library const { StorageTransferServiceClient, } = require('@google-cloud/storage-transfer'); /** * TODO(developer): Uncomment the following lines before running the sample. */ // Your project id // const projectId = 'my-project' // The agent pool associated with the POSIX data source. Defaults to the default agent // const sourceAgentPoolName = 'projects/my-project/agentPools/transfer_service_default' // The root directory path on the source filesystem // const rootDirectory = '/directory/to/transfer/source' // The ID of the GCS bucket to transfer data to // const gcsSinkBucket = 'my-sink-bucket' // Transfer manifest location. Must be a `gs:` URL // const manifestLocation = 'gs://my-bucket/sample_manifest.csv' // Creates a client const client = new StorageTransferServiceClient(); /** * Creates a request to transfer from the local file system to the sink bucket */ async function transferViaManifest() { const createRequest = { transferJob: { projectId, transferSpec: { sourceAgentPoolName, posixDataSource: { ``` -------------------------------- ### RunTransferJob Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 Starts a new operation for the specified transfer job. A transfer job can only have one active transfer operation at a time. ```APIDOC ## RunTransferJob ### Description Starts a new operation for the specified transfer job. A `TransferJob` has a maximum of one active `TransferOperation`. ### Method `rpc RunTransferJob(RunTransferJobRequest) returns (Operation)` ### Authorization Requires the following OAuth scope: `https://www.googleapis.com/auth/cloud-platform` ``` -------------------------------- ### Schedule JSON Structure Source: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs Specifies how transfers should be scheduled, including start and end dates, times, and repeat intervals for recurring jobs. ```json { "scheduleStartDate": { object (Date) }, "scheduleEndDate": { object (Date) }, "startTimeOfDay": { object (TimeOfDay) }, "endTimeOfDay": { object (TimeOfDay) }, "repeatInterval": string } ``` -------------------------------- ### Create and Run a Transfer Job (Go) Source: https://cloud.google.com/storage-transfer/docs/libraries This snippet demonstrates how to create and run a one-time transfer job between two Google Cloud Storage buckets using the Go client library. Ensure you have configured Application Default Credentials for authentication. ```go import ( "context" "fmt" "io" storagetransferpb "cloud.google.com/go/storage/transfer/apiv1" ) func createAndRunTransferJob(w io.Writer, projectID, sourceGCSBucket, sinkGCSBucket string) (*storagetransferpb.TransferJob, error) { ctx := context.Background() client, err := storagetransferpb.NewClient(ctx) if err != nil { return nil, fmt.Errorf("NewClient: %w", err) } defer client.Close() req := &storagetransferpb.TransferJob{ Project: projectID, TransferSpec: &storagetransferpb.TransferSpec{ ObjectConditions: &storagetransferpb.ObjectConditions{ LastModifiedBefore: nil, LastModifiedAfter: nil, }, TransferOptions: &storagetransferpb.TransferOptions{ DeleteObjectsFromSourceAfterTransfer: false, OverwriteObjectsAlreadyExistingInSink: false, TransferMode: storagetransferpb.TransferMode_TRANSFER_MODE_TRANSFER_ALL_TYPES, }, SourceSpeedLimit: &storagetransferpb.TransferSpec_BytesPerHour{BytesPerHour: 100000000000}, DestinationSpeedLimit: &storagetransferpb.TransferSpec_BytesPerHour{BytesPerHour: 100000000000}, Notification: nil, Status: storagetransferpb.TransferJob_ENABLED, Description: "", ClientTlsCertificates: nil, DataSource: &storagetransferpb.TransferSpec_GcsDataSource{ GcsDataSource: &storagetransferpb.GcsData{BucketName: sourceGCSBucket}}, DataSink: &storagetransferpb.TransferSpec_GcsDataSink{ GcsDataSink: &storagetransferpb.GcsData{BucketName: sinkGCSBucket}}, }, Status: storagetransferpb.TransferJob_ENABLED, } resp, err := client.CreateTransferJob(ctx, req) if err != nil { return nil, fmt.Errorf("failed to create transfer job: %w", err) } if _, err = client.RunTransferJob(ctx, &storagetransferpb.RunTransferJobRequest{ ProjectId: projectID, JobName: resp.Name, }); err != nil { return nil, fmt.Errorf("failed to run transfer job: %w", err) } fmt.Fprintf(w, "Created and ran transfer job from %v to %v with name %v", sourceGCSBucket, sinkGCSBucket, resp.Name) return resp, nil } ``` -------------------------------- ### Run On-Premises Agent with HTTPS Proxy Source: https://cloud.google.com/storage-transfer/docs/on-prem-agent-details?hl=id Use this command to run the agent, specifying an HTTPS proxy. Replace PROXY, PROJECT_ID, and ID_PREFIX with your specific values. Ensure PROXY uses an HTTP URL. ```bash sudo docker run -d --ulimit memlock=64000000 --rm \ --volumes-from gcloud-config \ -v /usr/local/research:/usr/local/research \ --env HTTPS_PROXY=PROXY \ gcr.io/cloud-ingest/tsop-agent:latest \ --enable-mount-directory \ --project-id=PROJECT_ID \ --hostname=$(hostname) \ --agent-id-prefix=ID_PREFIX ``` -------------------------------- ### ListAgentPoolsRequest Filter Example Source: https://cloud.google.com/storage-transfer/docs/reference/rpc/google.storagetransfer.v1 An optional query parameter to filter agent pools by name. Agent pool names must be specified as an array. ```json { "agentPoolNames":["agentpool1","agentpool2"] } ``` -------------------------------- ### Create a transfer job using a manifest file (Java) Source: https://cloud.google.com/storage-transfer/docs/manifest?hl=pt-BR This Java code snippet shows how to create and run a transfer job from a POSIX filesystem to a GCS bucket, using a manifest file to specify the source files. Ensure you replace placeholder variables with your actual project and bucket details. ```java import com.google.storagetransfer.v1.proto.StorageTransferServiceClient; import com.google.storagetransfer.v1.proto.TransferProto; import com.google.storagetransfer.v1.proto.TransferTypes.GcsData; import com.google.storagetransfer.v1.proto.TransferTypes.PosixFilesystem; import com.google.storagetransfer.v1.proto.TransferTypes.TransferJob; import com.google.storagetransfer.v1.proto.TransferTypes.TransferManifest; import com.google.storagetransfer.v1.proto.TransferTypes.TransferSpec; import java.io.IOException; public class TransferUsingManifest { public static void main(String[] args) throws IOException { // TODO(developer): Replace these variables before running the sample. // Your project id String projectId = "my-project-id"; // The agent pool associated with the POSIX data source. If not provided, defaults to the // default agent String sourceAgentPoolName = "projects/my-project-id/agentPools/transfer_service_default"; // The root directory path on the source filesystem String rootDirectory = "/directory/to/transfer/source"; // The ID of the GCS bucket to transfer data to String gcsSinkBucket = "my-sink-bucket"; // The ID of the GCS bucket which has your manifest file String manifestBucket = "my-bucket"; // The ID of the object in manifestBucket that specifies which files to transfer String manifestObjectName = "path/to/manifest.csv"; transferUsingManifest( projectId, sourceAgentPoolName, rootDirectory, gcsSinkBucket, manifestBucket, manifestObjectName); } public static void transferUsingManifest( String projectId, String sourceAgentPoolName, String rootDirectory, String gcsSinkBucket, String manifestBucket, String manifestObjectName) throws IOException { String manifestLocation = "gs://" + manifestBucket + "/" + manifestObjectName; TransferJob transferJob = TransferJob.newBuilder() .setProjectId(projectId) .setTransferSpec( TransferSpec.newBuilder() .setSourceAgentPoolName(sourceAgentPoolName) .setPosixDataSource( PosixFilesystem.newBuilder().setRootDirectory(rootDirectory).build()) .setGcsDataSink((GcsData.newBuilder().setBucketName(gcsSinkBucket)).build()) .setTransferManifest( TransferManifest.newBuilder().setLocation(manifestLocation).build())) .setStatus(TransferJob.Status.ENABLED) .build(); // Initialize client that will be used to send requests. This client only needs to be created // once, and can be reused for multiple requests. After completing all of your requests, call // the "close" method on the client to safely clean up any remaining background resources, // or use "try-with-close" statement to do this automatically. try (StorageTransferServiceClient storageTransfer = StorageTransferServiceClient.create()) { // Create the transfer job TransferJob response = storageTransfer.createTransferJob( TransferProto.CreateTransferJobRequest.newBuilder() .setTransferJob(transferJob) .build()); System.out.println( "Created and ran a transfer job from " + rootDirectory + " to " + gcsSinkBucket + " using " + "manifest file " + manifestLocation + " with name " + response.getName()); } } } ``` -------------------------------- ### Test Cloud Storage API Connectivity Source: https://cloud.google.com/storage-transfer/docs/on-prem-agent-details?hl=id Run this command to test connectivity to the Cloud Storage API from the agent's host machine. ```bash gcloud storage cp test.txt gs://MY-BUCKET ```